Divergence in gradient descent

I am trying to find a function h(r) that minimises a functional H(h) by a very simple gradient descent algorithm. The result of H(h) is a single number. (Basically, I have a field configuration in space and I am trying to minimise the energy due to this field). The field is discretized in space, and the gradient used is the derivative of H with respect to the field's value at each discrete point in space. I am doing this on Mathematica, but I think the code is easy to follow for non-users of Mathematica.

The function Hamiltonian takes a vector of field values, the spacing of points d, and the number of points imax, and gives the value of energy. EderhSym is the function that gives a table of values for the derivatives at each point. I wrote the derivative function manually to save computation time. (The details of these two functions are probably irrelevant for the question).

Hamiltonian[hvect_, d_, imax_] :=
Sum[(i^2/2)*d*(hvect[[i + 1]] - hvect[[i]])^2, {i, 1, imax - 1}] +
Sum[i^2*d^3*Potential[hvect[[i]]], {i, 1, imax}]

EderhSym[hvect_,d_,imax_]:=Join[{0},Table[2 i^2 d hvect[[i]]-i(i+1)d hvect[[i+1]]
-i(i-1)d hvect[[i-1]]+i^2 d^3 Vderh[hvect[[i]]], {i, 2, imax - 1}], {0}]

The code below shows a single iteration of gradient descent. hvect1 is some starting configuration that I have guessed using physical principles.

Ederh = EderhSym[hvect1, d, imax];

hvect1 = hvect1 - StepSize*Ederh;

The problem is that I am getting random spikes in the derivative table that grow and cause an overflow. Here is a plot of the derivative at the start, and after a few iterations (the number of iterations before overflow depends on the step size).  The spikes keep growing until there is an overflow. I have tried changing the step size, I have tried using moving averages, low pass filters, Gaussian filters etc. I still get spikes that cause an overflow. Has anyone encountered this? Is it a problem with the way I am setting up gradient descent?

Edit 1:

It seems to be working with a much smaller step size than I had previously tried (although convergence is slow). I believe that was the problem, but I do not see why the divergences are localised at particular points.

Edit 2:

I am testing my gradient descent code as I will have to adapt it to a different multivariable Hamiltonian where I do the following iteratively, to find a saddle point instead. (n is an appropriately chosen small number, 20 in my case). This makes it difficult to use adaptive step sizes. Otherwise, updating step sizes would be the best course of action.

For[c = 1, c <= n, c++,
Ederh = EderhSym[hvect1, d, imax];
hvect = hvect - d*StepSize*Ederh;
];

Ederh = EderhSym[hvect1, d, imax];
hvect1 = hvect1 + n*d*StepSize*Ederh;
• Double checking, you want to minimize: $$H(h,d,i_{max})=\frac{d}{2}\sum_{i=1}^{i_{max}-1}i^2[h(i+1)-h(i)]^2+d^3\sum_{i=1}^{i_{max}}i^2 V(h(i))?$$ If yes, what constraints do you have on $h$? Also what does $V$ look like? Is it singular near 450? – Alex R. Mar 31 '16 at 0:42
• V is smooth. It looks like this: i.imgur.com/QTYFg9z.png. h(r) is supposed to have zero slope at r = 0 and at r -> $\inf$. I also know the value of h(0). I used these to come up with a good starting configuration. – Gowri Mar 31 '16 at 1:32
• 1. Are you sure your analytical gradient is correct? Can you use a numerical differentiation routine (for once) to make sure you have the similar behaviour? 2. If you have a somewhat large step-size you will "jump around" near the optimum. Maybe you need to think of an adaptive step-size programme. 3. Going back to derivative: You computed using Mathematica right? Not by hand... 4. Maybe the divergence happened close to local minima. You do not show the values that caused, only the iteration number so we cannot really help. – usεr11852 Mar 31 '16 at 19:15
• 1. I had initially used Mathematica to find the derivative. I wrote the derivative function by hand to save computation time (as the symbolic derivative need not be computed every time). I get the same behaviour. 2. As I mentioned in the edit, I think step size is the problem, and with a small enough step, it seems to be going towards convergence. With a larger step size, random peas appeared and grew. I am trying to do this without using adaptive step sizes, as I am going to be modifying this to find a saddle point instead, and the algorithm I have in mind makes it hard to fix step size. – Gowri Mar 31 '16 at 21:34