Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
The text is from Intro to Statistical Learning Page no 380.Can anyone explain the both ideas clearly with an example if possible
1) In linear regression scaling has no effect.
2)In linear regression,multiplying a variable by a factor of c will simply lead to multiplication of the corresponding coefficient estimate by a factor of 1/c, and thus will have no substantive effect on the model obtained
It does not matter for fitted values and residuals if we change the units of measurement of $X$. Consider transforming $X $ by some invertible $k\times k$ matrix $A$, $XA$ (e.g., change months of schooling to years and meters to centimeters when explaining wages).
This is seen as follows, \begin{eqnarray*} P_{XA}&:=&XA\bigl((XA)'XA\bigr)^{-1}(XA)'\\ &=&XA\bigl(A'X'XA\bigr)^{-1}A'X'\\ &=&XAA^{-1}(X'X)^{-1}(A')^{-1}A'X'\\ &=&P_{X} \end{eqnarray*}
What about $\hat{\beta}$? Consider \begin{eqnarray*} \hat{\beta}^\circ&=&\bigl(\underbrace{A'X'}_{``X'"}\underbrace{XA}_{``X"}\bigr)^{-1}\underbrace{A'X'}_{``X'"}y\\ &=&A^{-1}(X'X)^{-1}(A')^{-1}A'X'y\\ &=&A^{-1}(X'X)^{-1}X'y\\ &=&A^{-1}\hat{\beta} \end{eqnarray*} That is, if $$ A=\begin{pmatrix} 1/12&0\\ 0&100 \end{pmatrix}\qquad\text{so that}\qquad A^{-1}=\begin{pmatrix} 12&0\\ 0&1/100 \end{pmatrix} $$ in the above example, the effect of a change in the regressors is, sensibly, adjusted accordingly.
