, 2 min read
Stiffness in Neural Networks
In below two posts I had already mentioned the close connection between neural networks and solving stiff ordinary differential equations:
- Neural Network Training using Stiff ODE Solvers
- Reply to: Neural Network Back-Propagation Revisited with Ordinary Differential Equations
Below threedimensional graph shaws different frequencies and when solving for the global maximum there are very many local maxima in the way.
$$
z = f(x,y) = e^{-(x^2+y^2)/25} \sin {\pi\over2}x \cos {\pi\over2}x
$$
In this case you want the stiff solver to ignore the high frequency solutions, but just follow the low frequency solution, thereby going straight to the global maximum.
The above graph is similar to an egg carton.
