Why normalize input variables in NN? I'm reading the 'Efficient Backprop' paper and it's mentioned that the reason to have a zero mean for the input variables is because otherwise the eigenvalue (for the hessian I think) will be very large. This would imply a large condition number so the cost surface will be steep in some directions and shallow in others leading to a slow convergence.
If the learning rates are independent for every weight (as is also recommended in the paper), then why should it matter if one direction in the cost surface is much steeper than another?
Edit: Note that the inputs are assumed to be linearly decorrelated.
Can anyone help?
 A: I think you're combining two concepts.


*

*Whitening. Even though your variables are decorrelated, they may not be whitened -- where each variable must have a variance of 1. This is slightly important (as you noticed) for convergence and conditioning. But it's very important if you're regularizing your weights, e.g. with L1 regularization to encourage sparsity. For two dimensions with very different variance to have similar influence, their weights will also be very different -- but the larger will be more penalized (and biased) due to the regularization. It's also very useful for interpreting weights, as they are already variance-adjusted; larger weights indicate more informative input dimensions.

*De-meaning. This is necessary if you aren't optimizing each neuron's output threshold / input bias. Even if you are, demeaning starts the optimization with more neutral initial conditions. If you look at the sigmoid, an input with mean 100 and variance 1 would (virtually) never be able to produce an output below 0.5, no matter what the weight (assuming no bias or threshold).
