Multiplying a predictor by a constant in Lasso/Ridge regression

Question

If we multiply one of predictors by a constant $c$ in the regression set-up for all data points. What happens to the weights (or specifically weight corresponding to that predictor) if we are doing Lasso/Ridge regression?

In the case of OLS it is clear the effect would be the corresponding weight will be multiplied by $1/c$. Now if we assume $c=2$ and let's assume we are doing Lasso. If we multiply that corresponding weight by $1/2$ the mean squared part of loss function doesn't change and the $L^1$ part decreases. So you can say the minima of the new loss function will be less than the older one. But I'm not sure whether this implies anything about the individual coefficients or not. This might also depend whether $c>1$ or $c<1$.

Answer 1

The usual practice in penalized methods like LASSO, ridge or principal-component regression is to start by standardizing all predictors to zero mean and unit standard deviation. Otherwise the issue that you raise would lead to a distance measured in miles being handled differently than the same distance measured in millimeters. That's the default, for example, in the R glmnet package, with coefficients then re-corrected silently after the penalization to the original predictor values. In that case, once you get coefficients in the original scales you can treat them as you would with ordinary least squares.