It is well known that in Ridge or LASSO regression we add a regularization term to penalize large regression coefficients. What if the true relationship between the response and covariates relies on a large coefficient? Let's say, the true relation is given by $y=2.5x_1+1.5x_2+200x_3$, where the third term will get unfairly penalized in regression for its large but true coefficient. One way to avoid that is to standardized the data. But is that always necessary? Thanks in advance!
2
-
$\begingroup$ @JanKukacka oddly enough the highly popular page you linked says almost nothing about regularized (as opposed to standard linear) regression, except in one comment! (I was surprised when I looked back at that page.) This page seems to be more on point. $\endgroup$– EdMCommented Mar 20, 2019 at 14:25
-
$\begingroup$ @EdM Fair point. I nominated it as a duplicate based on a now-removed answer by another user. $\endgroup$– Jan KukackaCommented Mar 20, 2019 at 14:55
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
This opens up a big topic that is addressed in the maximum likelihood estimation chapter in my book Regression Modeling Strategies. And the links provided by others above do not address penalization.
To put this into perspective, note that you will not know the true regression coefficients in practice. But if you knew that one coefficient is likely to be larger than the others, you could elect not to penalize that coefficient at all, or to put a milder penalty on that one. In the Bayesian world this would be called putting a less skeptical prior on the one with the suspected large coefficient.
Directly to your question, standardizing the data mainly helps if the predictors have different scales. It can also help, even if their scales were the same, in choosing the ridge, lasso, elastic net, etc. penalty parameter(s).
So you need to distinguish between different scales and different effects on $Y$.
Synonymous with the need to specify a variance in a prior in Bayesian modeling, penalized maximum likelihood estimation has a scaling problem when scales of the $x$'s differ, and it is traditional to scale by the $x$ standard deviations. This is arbitrary but usually better than not scaling. In my book I make the model a little more interpretable by putting the scaling constants in the penalty function instead of pre-processing $x$'s. This is also the way my R rms package works for penalizing logistic and ordinary regression models.
answered Mar 20, 2019 at 11:49
$\begingroup$
$\endgroup$
Lasso and Ridge Regression "shrinks" the entire vector of coefficients. It is used so the less significative ones are zero (lasso) or near zero (ridge). So in your example the larger coefficient will remain and the other two are going to zero, reflecting the importance of $x_3$ in relation to $y$ (assuming that $x_1, x_2$ and $x_3$ have similar units)
$\begingroup$
$\endgroup$
its not exactly unfair: the lasso parameter specifies a value in error reduction per unit change in coefficient. ie if w3 has a much larger impact on the error than w2 it will not be significantly impacted.
the ideal case is vice versa - you know the relative scales of your inputs and you leave them unchanged. essentially the regularisation is enforcing small changes in the input variable should have small change in output. If you are a scientist, hopefully you already have a scale for each variable, if you don't then the default choice is to use the standard deviation as a scale.
Take for example exam scores for predicting eg degree result, you don't know apriori what's a good score, so normalising the scores is a good assumption ( ie that the distribution of good and bad stays roughly the same across different exams)
