13
$\begingroup$

I know for LASSO and elastic net regression, standardization is important, because coefficient penalization in regularization will be biased if the ranges of data are different.

Meanwhile, OLS regression does not run into this problem. We are just using stochastic gradient descent to find the best coefficients to minimize error. Maybe it would impact it if the step size or learning rate biases against larger ranged columns, such that you're not moving across as large of a range of the gradient for those columns? Maybe it's also that there's a closed-form solution for OLS, so even that issue with stochastic gradient descent doesn't matter?

So, given neural networks are essentially like multiple layers of regressions, why is data standardization recommended or more common practice? Is it because there's more layers of regressions, so standardizing just makes it more stochastic gradient descent more efficient, while efficiency to getting to a solution isn't really a problem in OLS?

$\endgroup$

4 Answers 4

12
$\begingroup$

In the backpropagation algorithm used to train neural networks, the update is proportional to the derivative of the activation function. If a hidden layer neuron is saturated at +/-1, the derivative will be zero and the associated weights will no longer be updated. The reason for standardisation of the inputs is to help ensure that the input layer weights can be safely initialised to small random values such that none of the first hidden layer units are likely to be saturated and permanently stuck at +/-1 (and hence not contributing anything). The initialisation of weights in higher layers is less problematic as the range of their inputs is governed by the activation function used in the preceding layer.

It also matters if regularisation (weight decay) is being used. Scaling the input by 2 will give the same functional behaviour if the corresponding input layer weight is scaled by 0.5. However that input will be less regularised because the natural value for the weight is lower for the same functional behaviour. So it also makes the effects of regularisation more consistent for different inputs if they are standardised. Effectively, if you don't standardise the inputs, some will be whispering at the network and others will be SHOUTING, just like LASSO and Elastic Net.

BTW L1 regularisation (i.e. LASSO) was used quite early in neural networks, see Williams (1993), perhaps earlier than it was in statistics (initially invented earlier still in geophysics)?

$\endgroup$
9
$\begingroup$

In cases where the Ordinary Least Squares (OLS) has a single solution, and given that you have a suitable learning rate, gradient descent will always take you towards it, whether it is too small or big in some dimensions. Though, this situation could still slow down the learning, and makes you more vulnerable because that suitable learning rate is harder to choose.

In neural networks, this is not the case. Depending on the size, it may have hundreds of thousands of local minima, maybe even more gradient plateaus and you may end up anywhere at the end if your features do not behave well enough. Not to mention that these plateaus are partly constructed by the activations in neural networks, which is in line with your comment that it's multiple layers of regressions (with activations in-between).

In practice, there may still be numerical issues, however, in OLS depending on your feature ranges, and the effective range of your chosen data type (e.g. you can't tackle numbers like $2e+1000$ using a float data type), or the gradients will not behave very well, because of these numerical anomalies. In that case, maybe your only weapon is to standardize.

Apart from the case where the two uses the same optimization algorithm, OLS has more options because the optimization problem is much more simpler, like second-order methods (newton's method), cholesky decomp etc. which are not available to solving general neural nets (yet).

$\endgroup$
7
$\begingroup$

When doing standard OLS the parameter estimates are not penalized (shrunk; regularized). As has been so well described above, in this case it doesn't matter if you standardize or not. Interpretation is usually easier if you don't standardize. Once you shift into penalized regression (e.g., ridge or lasso) it is very difficult to choose penalty parameters on the original data scale, and standard practice, though not always justified, is to scale predictors by their standard deviations (even if the predictors have an asymmetric distribution, which is problematic) to make them unitness and to make it more likely that a single penalty parameter (two if using elastic net) can possibly apply to predictors of different scales.

I prefer to put the scaling constants into the penalty function itself so that I can interpret all the coefficients on the original scales. See this.

$\endgroup$
1
  • 1
    $\begingroup$ While this is useful information, it doesn't seem to address the Question at all. The asker has explicitly noted the exception of penalized regression and wants to know about unpenalized OLS vs neural nets. $\endgroup$ Commented Aug 30, 2023 at 21:13
2
$\begingroup$

To add one aspect to the other answers (I'm not discussing the neural networks case in any depth so for that look elsewhere): OLS regression can easily be shown to be affine equivariant. This means that linear operations on the data, including standardising the variables, will produce equivalent solutions; it just doesn't make a difference theoretically.

For neural networks it will depend on how exactly they are set up, see other answers. (Dikran Marsupilami in the comments makes the case that for unregularised networks it is possible to initialise them in order to be invariant against standardising.)

$\endgroup$
10
  • 2
    $\begingroup$ For an MLP neural network, if you perform an affine transformation of the inputs there will be an affine transformation of the first layer weights such that the network will be functionally equivalent (and have the same unregularised training loss), so the distinction doesn't seem clear to me. Am I missing something? $\endgroup$ Commented Aug 27, 2023 at 11:23
  • $\begingroup$ @DikranMarsupial Doesn't your own answer explain that it's not fully functionally equivalent? If it were, why would standardisation make a difference? $\endgroup$ Commented Aug 27, 2023 at 12:49
  • $\begingroup$ @ChristianHenning that is only where regularisation is used (and the situation the same as for LASSO or ridge regression). For the hidden layer units being stuck at +/-1 the affine transformation would still leave them stuck at +/-1. $\endgroup$ Commented Aug 27, 2023 at 12:52
  • 1
    $\begingroup$ @DikranMarsupial But if it matters in the first (non-regularisation) case, you don't have "functional equivalence". Anyway, I'm not the best expert for neural networks so I shouldn't make general statements about them. Will add something to make it seem less general what I claim there. However I'm still puzzled about the fact that in one place you say that standardisation matters in non-regularisation situations whereas in another place you say it only matters for regularisation. $\endgroup$ Commented Aug 27, 2023 at 13:03
  • 1
    $\begingroup$ It matters for unregularised models for at least one reason, it matters for regularised models for at least two reasons. $\endgroup$ Commented Aug 27, 2023 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.