Why don't neural networks get stuck in loops when they overshoot a backprop step? In a normal feedforward network I wrote with linear activations I've noticed that after a while when the network has found a pretty viable solution to a problem it sometimes takes a step in the wrong direction. At this point, the learning rate is very small but the steps still causes a noticeable change in the error. What appears to follow is quite puzzling, the error, weights and biasses all grow very until eventually, they become too big and the network breaks.
My hypothesis is that when it takes a step in the wrong direction the error goes up and so does the learning rate. These two factors combine so that when the weights and biasses are adjusted to take a step in the opposite direction they are the derivatives are larger and so are the adjustments. For this reason on the way back in the other direction, it overshoots even further than the time before, ultimately creating an infinite loop causing the error, weights, biasses and learning rate to grow exponentially. (I've got a cap on the learning rate so it maxes out at like 0.05).
This, however, is just a hypothesis. If this is possible how is it normally handled... If this isn't it and I've just made a mistake why doesn't this happen? 
Thanks for your help in advance.
 A: Your hypothesis sounds right to me.
Edit:
What bothers me in your hypothesis is the notion of the increasing learning rate. Usually the backpropagation diverges due to a big(ish) learning rate. If that's the case you should try smaller learning rate.
So I would state it more like:


*

*The learning rate is not small enough so when the backpropagation goes in the wrong direction, the network's correction is too big due to the large learning rate so when it tries to "go back" in the right direction it actually "overshoots" and goes further than it should have. This repeats, and you observe the divergence of the network.


Key point: try using smaller learning rate.
Original answer: 
You are basically describing divergence. I think it happens mostly when you have a large learning rate.  I got a little confused how you use the learning rate exactly, but its main idea is to limit the the delta with which the weights (and biases) are updated. So if you have large learning rate the updates of the weights would be significant. I think this is the main cause for divergence.
To handle it try starting with smaller learning rate, e.g. 0.01, 0.001, 0.0001 - do some experiments. You also mention that you change the learning rate. If it's easy to change that - try running the net with a fixed (smaller) learning rate.
You could also tune the initialisation of the weights if you haven't already. Check if the "He et al initialization" would help with this. I believe you already initialise them randomly so I don't think this could cause the divergence - it should just speedup the learning. 
