0
$\begingroup$

The title says it all: How can I see that OLS estimators (linear regression) do not depend on the scale of the predictor data? I am reading my lecture notes about lasso estimators, where it is mentioned that lasso estimators do depend on the scale of the predictor data, whereas OLS estimators don't. I can somehow see why this would be true for lasso estimators (we need to constrain the L1-norm of the coefficient estimators) and reason why it would be true for OLS estimators (the constraint disappears), but how can I understand this without considering lasso estimators and their properties first?

Thanks in advance!

$\endgroup$
1
$\begingroup$

OLS simply works with whatever units you provide in the data. The contribution of predictor $i$ to the linear predictor is $\beta_i x_i$. Any changes in the scale in which $x_i$ is expressed will just result in a proportional change in the value returned for $\beta_i$.

For example, say one of your predictors is a distance, measured in miles, and the outcome is a cost expressed in dollars. The coefficient in OLS would be in units of dollars per mile.

That distance could also be measured in millimeters, with 1609344 millimeters per mile. If you instead do the OLS in those units, the coefficient would be dollars per millimeter.

As the fundamental physical reality doesn't depend on the units of the predictor, the coefficient expressed in dollars per mile would be exactly 1609344 times the coefficient expressed in dollars per millimeter. The fundamental model doesn't depend on the units of the predictor, although the numeric values of the coefficients will differ proportionately to the units.

As methods like LASSO penalize the numeric values of the regression coefficients, this poses a problem. If you use a distance measured in miles, its regression coefficient will be 1609344 times larger than if you measured it in millimeters, and the penalization would depend massively between the two measurement units. Standard practice is thus to start by standardizing continuous predictors by subtracting the mean and dividing by the standard deviation, to obtain predictor values without units. Once penalization is done, you can go back to the original units, or re-express in any other choice of units.

$\endgroup$
2
$\begingroup$

Try an algebraic example. The OLS estimator is:

$$ \hat{\beta}_{OLS} = (X^TX)^{-1}X^Ty $$

Now scale $X$ and $y$ by a scalar $\textrm{a}$:

$$ \hat{\beta}^{scaled}_{OLS} = ((aX)^T(aX))^{-1}(aX)^T(ay) \\ =(X^TX a^2)^{-1}X^Tya^2 \\ =(X^TX)^{-1}X^Ty \underbrace{\frac{a^2}{a^2}}_{=1} = \hat{\beta}_{OLS} $$

Note that if only $X$ is scaled, then the estimator is scaled by the same factor $\hat{\beta}^{scaled}_{OLS}=a\hat{\beta}_{OLS}$ (since you would have $a^2/a=a$ in the last step).

$\endgroup$
1
  • $\begingroup$ Wow, this was surprisingly easy to verify mathematically. Thank you! $\endgroup$ – Zachary Nov 22 '20 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.