Lasso centering and standarization with R

Question

I am working with a lasso regression with the glmnet package. I read these threads: When conducting multiple regression, when should you center your predictor variables & when should you standardize them?, Need for centering and standardizing data in regression and Is standardisation before Lasso really necessary?.

Based on the responses I decided that I need to standardize my data before using it. I do have some questions however:

• Do I need to standardize the predictors and the responses or only the predictors?
• I am using the function scale(myData, center = TRUE, scale = TRUE) for building the model, but I am wondering what do I do when I want to do predictions with a test data set. I think I should also standardize and center the test data, but how to I do that? Substracting the mean from the initial (training) dataset and the dividing it by the standard deviation of the initial dataset?
• When I get a result do I need to "backscale" it (using the original mean and standard deviation) or do I already get the "final" result?

Answer 1

If you use glmnet, the scaling is performed by the package. You don't need to worry about scaling the test set because the "coefficients are always returned on the original scale".

By default:

glmnet(x, y, [...]
standardize = TRUE,
intercept = TRUE,
standardize.response = FALSE [...])

As for the standardization of the response, it should not change the performance of your model after cross validating over $\lambda$ so you can set standardize.response = FALSE

Indeed the LASSO solves

$$ \min_\beta\; \| Y - X\beta \|^2_2 + \lambda \|\beta\|_1 $$

Scaling $Y$ by a factor $\alpha > 0$, the problem becomes

$$ \min_\beta\; \| \alpha Y - X\beta \|^2_2 + \lambda \|\beta\|_1 $$

which is equivalent to

$$ \min_\beta\; \alpha \| Y-X\beta/\alpha \|^2_2 + \lambda \|\beta\|_1 $$

$$ \min_\beta\; \| Y - X\beta/\alpha \|^2_2 + \lambda \|\beta/\alpha\|_1 $$

So it has the same value of $\lambda$

Answer 2

In general, you are right to worry about scaling the responses. If you optimize a function of the kind that LASSO is based on,

$$ \min_{\beta} || Y - X\beta ||_{2}^{2} + \lambda || \beta ||_1, $$

then scaling the response $Y$ with some constant $\alpha$,

$$ \min_{\beta} || \alpha Y - X\beta ||_{2}^{2} + \lambda || \beta ||_1 $$

leads to

$$ \min_{\beta} \alpha^2 || Y - X\beta / \alpha ||_{2}^{2} + \lambda || \beta ||_1. $$

Notice the square on the $\alpha$, which is missing in the accepted answer. Dividing by $\alpha^2$ leads to $$ \min_{\beta} || Y - X\beta / \alpha ||_{2}^{2} + (\lambda / \alpha) || (\beta / \alpha) ||_1, $$

which describes a solution that will perform differently than a solution of the original problem since effectively the regularization constant changed.

However, if you look at the documentation for glmnet, you will find that the default behavior is to use a log spaced grid for $\lambda$ and choose a range for the grid based on standardized (!) responses and predictors.

So the answer to your first question is, based on the glmnet documentation: If you use default values for all parameters, standardizing the response should not have a big effect on performance since the $\lambda$ grid is chosen using standardized responses anyway. For non-default parameters (e.g., using less values in the $\lambda$ grid by setting nlambda to a low value), it might have a larger effect and you have to be careful. Also, this does not carry over to, e.g., sklearn, where no grid is used and changing the response scale might drastically affect performance.

The answer to your second and third questions is that by default, the package standardizes the data before using it and rescales the coefficients before returning them.

Answer 3

With a lasso regression, standardization is essential. That's because lasso finds the best solution subject to a constraint on the absolute value of the sum of the coefficients. If one didn't scale the coefficients the answer would totally depend on the scaling of the coefficient. For example using lasso on $x_1, x_2 $ as opposed to $x_1, y=\frac{1}{10000} x_2$ would give very different answers. With the second set of variables, the coefficient of y is almost guaranteed to be zero with lasso. Check the glmnet help, I seem to recall that it will automatically scale the data