5
$\begingroup$

I am doing linear regression with multiple features/variables. I decided to use normal equation method to find coefficients of linear model. If we use gradient descent for linear regression with multiple variables we typically do feature scaling in order to quicken gradient descent convergence. For now, I am going to use normal equation method with formula:

$$\hat{\beta} = (X^TX)^{-1}X^Ty = X^+y$$ Source: The normal equations (Andrew Ng lecture notes, p. 11)

I have two contradictory information sources. In first it is stated that no feature scaling required for normal equations. In another I can see that feature normalization has to be done.

Sources:

  1. http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&doc=exercises/ex3/ex3.html
  2. http://puriney.github.io/numb/2013/07/06/normal-equations-gradient-descent-and-linear-regression/

At the end of these two articles information concerning feature scaling in normal equations presented.


The question is: do we need to do feature scaling (normalization) before normal equation analysis?

$\endgroup$
5
$\begingroup$

Well, in the second article there is a sentence:

Note that before conducting linear regression, you should normalize the data. One way is $\frac{x_i−mean(x)}{Range(x)}$, and some use $sd(x)$ as the denominator. Both work.

But is not said that it applies specifically to normal equations. And in gradient descent section there is nothing said about normalization. So I suppose it was a small mistake to include that sentence in Normal Equation section instead of Gradient Descent.

Anyway Andrew Ng is pretty authoritative on machine learning topic, so you can rely on his words:

Using this formula does not require any feature scaling, and you will get an exact solution in one calculation: there is no 'loop until convergence' like in gradient descent.

$\endgroup$
  • 1
    $\begingroup$ Is there any citable source of this statement of Andrew Ng? I remember this statement of the Coursera Machine learning online course. But I couldn't find this statement for example in the old CS 229 lecture notes: cs229.stanford.edu/materials.html $\endgroup$ – Luke Jul 26 '16 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.