# Why glmnet's predict function using a ridge regression fit returns different results with standardize=T vs standardize=F?

I want to do prediction using a ridge regression fit using the glmnet package. There are 2 options of the standardize parameter; 1) standardize=T (which is the default) and 2) standardize=F. I was thinking standardization of the data prior to regression is just for interpretation of the coefficients and it would not affect the prediction result. However, that's not what I see using the below example code (pred1 and pred2 are different from each other).

varsize = 50
samsize = 20
lambda = 1

set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps

trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]

library(glmnet)

pred1 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0, standardize=F), newx=testX)

print(cbind(pred1, pred2))


And the print result is:

 [1,]  4.0063487  3.8358423
[2,] -0.2935879 -0.5569798
[3,]  0.8406129  0.3931324
[4,] -3.0215173 -1.7771050
[5,] -2.0329757 -4.1598014
[6,]  0.1703848 -0.3710678
[7,] -3.3436552 -3.2583682
[8,] -3.3409509 -2.6274531
[9,] -3.2172309 -4.0190585
[10,] -0.6965232 -0.3061931


However, with a linear regression, I would get the same result for pred1 and pred2. Here is the example where just the $\lambda$ value is set to 0 in glmnet call (so fitting a linear regression model rather than a ridge regression model):

varsize = 50
samsize = 20

set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps

trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]

library(glmnet)

pred1 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0, standardize=F), newx=testX)

print(cbind(pred1, pred2))


And here is the result:

 [1,]  -0.9765420  -0.9765420
[2,]   6.7416965   6.7416965
[3,]   3.6099447   3.6099447
[4,]  -8.9710734  -8.9710734
[5,]   4.0057033   4.0057033
[6,]   6.9094331   6.9094331
[7,] -10.6853907 -10.6853907
[8,] -17.1785219 -17.1785219
[9,]  -5.6884246  -5.6884246
[10,]  -0.6812534  -0.6812534


Why is this behavior different in ridge regression? Why does standardization affect the result?

• The documentation explicitly says not to set lambda to a single constant. Feb 8, 2017 at 2:13
• This was just for example purposes. I choose lambda by cross validation in the actual code. Feb 8, 2017 at 6:36
• Sure, but when the documentation explicitly says not to do something, you shouldn't do it even in an example. If you are misusing the library, there is no reason to trust the results. Feb 8, 2017 at 16:36
• That's not the point here, I guess. The library suggests not to supply a single lambda value because the user is never able to know which lambda value would give a reasonable or meaningful result, so checking for a series of lambda values and choosing the one with the most meaningful result (smallest error for example) is what one should do. But here, I am not interested in the result itself at all, I am just interested in comparison of it with the result when a parameter value is set to a different option. So, a single lambda value would be completely fine here. Feb 8, 2017 at 22:19
• You will also observe drastic differences when standardizing the test set (eg. using scale()) during prediction. In general, I believe it is best to perform standardization externally and disable the default glmnet standardization. Oct 30, 2020 at 17:12

A toy example shows why standardized and non-standardized calculations will produce different coefficients with penalization. Say that you had a set of data in which exactly $y=x$, with $y$ in grams and $x$ in mm, with the empirical distribution of $x$ found to be $N$(0,1 mm). Then the coefficient that provides a perfect fit would be 1 gram/mm, and the penalty term in ridge regression at that perfect fit would be equal to $\lambda$. The final penalized coefficient would trade off the perfect fit against that penalty.
If you instead measured $x$ in centimeters but standardize $x$, you get the same tradeoff of fit against penalty as above. If you don't standardize in this case then you can still get a perfect fit to the data with a coefficient of 10 (now gram/cm), but the penalty in ridge regression to tradeoff against that is now $100\lambda$. No reason to suspect that the tradeoffs and the final choice of coefficients would be the same in the 2 cases. In your example, I suspect that differences between the empirical distributions and the normal distributions from which you sampled led to analogous, although less dramatic, differences in the penalized coefficients.