Do extra hidden layers prevent convergence? I have designed a simple feed-forward neural network using stochastic gradient descent. I use 22 inputs, 4 hidden layers, 1 output and am using a learning rate of 0.7 and momentum of 0.3. I have about 700 points in my training set. 
As I began training, I noticed that my MSE (calculated on a validation set of about 200 points) was increasing with each epoch. I then decided to reduce the number of hidden layers I was using from 4 to 3. This - for whatever reason - worked, and the MSE dropped with each epoch and the network converged. 
This leads me to lack confidence in my implementation of the backpropagation algorithms.
Why would this occur? Do extra, unneeded hidden layers prevent convergence? Could it be my algorithm?
 A: Training deeper networks becomes harder as you can incur into the Vanishing Gradient Problem, especially if the activation function you are using is the sigmoid.
If you look at the derivative of the sigmoid function you will see that it's bell-shaped:
This means that there is plenty of possibilities for the gradient of the sigmoid to be very low. 
The gradients of the parameters of earlier layers are calculated via multiplication with the gradients of further layers. If the gradients are typically less than 1, you can see how, as you add more layers, the gradient of earlier layers gets pushed towards 0. This is explained in detail from Michael Nielsen's Neural Networks and Deep Learning book (chapter 5).
One solution would be to use alternative activation functions, such as ReLu, or Exponential Linear Unit, which do not have low gradients for large inputs.
Another "solution" would be to just employ a much larger number of epochs.
Moreover, make sure that proper initialization of the weights (Xavier) is employed.
A: If your error function diverges try reducing your learning rate from 0.7 to say 0.007 or 0.07. More or less hidden layers should not affect convergence though the generalization power of the two would be different.
A: More hidden layers shouldn't prevent convergence,  although it becomes more challenging to get a learning rate that updates all layer weights efficiently. However, if you are using full-batch update, you should be able to determine a learning rate low enough to make your neural network progress or always decrease the objective function.
Assuming that you are using full-batch update, at a given iteration, in order to guarantee sufficient objective function decrease without manually specifying a learning rate, you can perform line search to find a learning rate that satisfies the two Wolfe conditions. 
You can also use L-BFGS with line search to optimise your neural network efficiently. L-BFGS uses an approximation of the Hessian (second order gradient) which in a way sets a learning rate for every parameter. minFunc for Matlab and scipy.optimise for python have L-BFGS.
However, before going further I would do the following two things:
1) I would check that the implementation is correct by checking that the gradient function is correct. This link 1 shows a method for checking that the gradient function is correct. First, numerically approximate the gradient for a parameter using the function value, then compare it with the value returned by the gradient function. The two values should be approximately the same.
2) I would also compare the neural network (NN) results with easy-to-use NN code available online. The autograd python library shows a quick implementation of multi-layer perceptron and its performance on a toy example. You can set the learning rate and the number of hidden layers fairly easy.
Hope this helps!
A: For a deep neural network that you mention, finding an effective local minima is the key. 
As per the paper, Gülçehre, Çağlar, and Yoshua Bengio. "Knowledge matters: Importance of prior information for optimization." Journal of Machine Learning Research 17.8 (2016): 1-32.
It says

Deeper Networks are Harder Hypothesis: Although solutions may exist,
  effective local minima are generally more likely to hamper learning as
  the required depth of the architecture increases.

