In Machine Learning I understand that feature scaling or mean-normalizing your input is useful when trying to learn a model using gradient descent. It's useful in the sense that it speeds up training as the "contour lines" are not elliptical, which makes it easier to find an optimal point in the parameter space. Andrew Ng explains the logic here.

However, if your data is "simple" enough so that you can solve it in closed-form, what's the point of normalization? Figuring out the weights $w$ that minimize sum-of-squares error is trivial:

$$w=(X^T X)^{-1} X^T Y$$

Is it safe to say normalization doesn't change anything? The speed is the same (same number of steps) and the results are the same. What's the justification for normalization? Or am I wrong?

I've compared the errors on normalized and non-normalized data in polynomial regression. For the best fit model, the errors are the same. Computation time is the same (actually there is a minuscule increase for the normalization step). Errors for polynomial degrees far from the best fit, however, are vastly different between normalized and non-normalized. For these, the error on the normalized set are actually very high. In this toy example, $x$ only has one feature and there are 100 samples in the data set.

  • 1
    $\begingroup$ You are correct, in closed form solution there is no need to normalize the features. Can you quote a source where you read this, Andrew Ng himself had stated in an exercise you don't need to normalize for closed form solution. openclassroom.stanford.edu/MainFolder/… $\endgroup$ – Amitoz Dandiana Sep 26 '16 at 17:23
  • $\begingroup$ Interesting. For one of my own courses I am asked to normalize input when solving with closed-form, which I thought did not add any value. $\endgroup$ – vega Sep 26 '16 at 17:41
  • $\begingroup$ I just answered a very similar question here: stats.stackexchange.com/questions/237039/… You may find that answer helpful. $\endgroup$ – Matthew Gunn Sep 26 '16 at 20:32
  • $\begingroup$ It depends on the fitting method used. For instance, when using regularization ... See my answer here: stats.stackexchange.com/questions/231285/… $\endgroup$ – kjetil b halvorsen Feb 21 '18 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.