I've always learned that data standardization is not necessary for OLS regression, but then recommended for neural networks. Intuitively, why is that?

Question

I know for LASSO and elastic net regression, standardization is important, because coefficient penalization in regularization will be biased if the ranges of data are different.

Meanwhile, OLS regression does not run into this problem. We are just using stochastic gradient descent to find the best coefficients to minimize error. Maybe it would impact it if the step size or learning rate biases against larger ranged columns, such that you're not moving across as large of a range of the gradient for those columns? Maybe it's also that there's a closed-form solution for OLS, so even that issue with stochastic gradient descent doesn't matter?

So, given neural networks are essentially like multiple layers of regressions, why is data standardization recommended or more common practice? Is it because there's more layers of regressions, so standardizing just makes it more stochastic gradient descent more efficient, while efficiency to getting to a solution isn't really a problem in OLS?

Answer 1

In the backpropagation algorithm used to train neural networks, the update is proportional to the derivative of the activation function. If a hidden layer neuron is saturated at +/-1, the derivative will be zero and the associated weights will no longer be updated. The reason for standardisation of the inputs is to help ensure that the input layer weights can be safely initialised to small random values such that none of the first hidden layer units are likely to be saturated and permanently stuck at +/-1 (and hence not contributing anything). The initialisation of weights in higher layers is less problematic as the range of their inputs is governed by the activation function used in the preceding layer.

It also matters if regularisation (weight decay) is being used. Scaling the input by 2 will give the same functional behaviour if the corresponding input layer weight is scaled by 0.5. However that input will be less regularised because the natural value for the weight is lower for the same functional behaviour. So it also makes the effects of regularisation more consistent for different inputs if they are standardised. Effectively, if you don't standardise the inputs, some will be whispering at the network and others will be SHOUTING, just like LASSO and Elastic Net.

BTW L1 regularisation (i.e. LASSO) was used quite early in neural networks, see Williams (1993), perhaps earlier than it was in statistics (initially invented earlier still in geophysics)?

Answer 2

In cases where the Ordinary Least Squares (OLS) has a single solution, and given that you have a suitable learning rate, gradient descent will always take you towards it, whether it is too small or big in some dimensions. Though, this situation could still slow down the learning, and makes you more vulnerable because that suitable learning rate is harder to choose.

In neural networks, this is not the case. Depending on the size, it may have hundreds of thousands of local minima, maybe even more gradient plateaus and you may end up anywhere at the end if your features do not behave well enough. Not to mention that these plateaus are partly constructed by the activations in neural networks, which is in line with your comment that it's multiple layers of regressions (with activations in-between).

In practice, there may still be numerical issues, however, in OLS depending on your feature ranges, and the effective range of your chosen data type (e.g. you can't tackle numbers like $2e+1000$ using a float data type), or the gradients will not behave very well, because of these numerical anomalies. In that case, maybe your only weapon is to standardize.

Apart from the case where the two uses the same optimization algorithm, OLS has more options because the optimization problem is much more simpler, like second-order methods (newton's method), cholesky decomp etc. which are not available to solving general neural nets (yet).

Answer 3

When doing standard OLS the parameter estimates are not penalized (shrunk; regularized). As has been so well described above, in this case it doesn't matter if you standardize or not. Interpretation is usually easier if you don't standardize. Once you shift into penalized regression (e.g., ridge or lasso) it is very difficult to choose penalty parameters on the original data scale, and standard practice, though not always justified, is to scale predictors by their standard deviations (even if the predictors have an asymmetric distribution, which is problematic) to make them unitness and to make it more likely that a single penalty parameter (two if using elastic net) can possibly apply to predictors of different scales.

I prefer to put the scaling constants into the penalty function itself so that I can interpret all the coefficients on the original scales. See this.

Answer 4

To add one aspect to the other answers (I'm not discussing the neural networks case in any depth so for that look elsewhere): OLS regression can easily be shown to be affine equivariant. This means that linear operations on the data, including standardising the variables, will produce equivalent solutions; it just doesn't make a difference theoretically.

For neural networks it will depend on how exactly they are set up, see other answers. (Dikran Marsupilami in the comments makes the case that for unregularised networks it is possible to initialise them in order to be invariant against standardising.)