Why glmnet's predict function using a ridge regression fit returns different results with standardize=T vs standardize=F?

Question

I want to do prediction using a ridge regression fit using the glmnet package. There are 2 options of the standardize parameter; 1) standardize=T (which is the default) and 2) standardize=F. I was thinking standardization of the data prior to regression is just for interpretation of the coefficients and it would not affect the prediction result. However, that's not what I see using the below example code (pred1 and pred2 are different from each other).

varsize = 50
samsize = 20
lambda = 1

set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps

trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]

library(glmnet)

pred1 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0, standardize=F), newx=testX)

print(cbind(pred1, pred2))

And the print result is:

 [1,]  4.0063487  3.8358423
 [2,] -0.2935879 -0.5569798
 [3,]  0.8406129  0.3931324
 [4,] -3.0215173 -1.7771050
 [5,] -2.0329757 -4.1598014
 [6,]  0.1703848 -0.3710678
 [7,] -3.3436552 -3.2583682
 [8,] -3.3409509 -2.6274531
 [9,] -3.2172309 -4.0190585
[10,] -0.6965232 -0.3061931

However, with a linear regression, I would get the same result for pred1 and pred2. Here is the example where just the $\lambda$ value is set to 0 in glmnet call (so fitting a linear regression model rather than a ridge regression model):

varsize = 50
samsize = 20

set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps

trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]

library(glmnet)

pred1 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0, standardize=F), newx=testX)

print(cbind(pred1, pred2))

And here is the result:

 [1,]  -0.9765420  -0.9765420
 [2,]   6.7416965   6.7416965
 [3,]   3.6099447   3.6099447
 [4,]  -8.9710734  -8.9710734
 [5,]   4.0057033   4.0057033
 [6,]   6.9094331   6.9094331
 [7,] -10.6853907 -10.6853907
 [8,] -17.1785219 -17.1785219
 [9,]  -5.6884246  -5.6884246
[10,]  -0.6812534  -0.6812534

Why is this behavior different in ridge regression? Why does standardization affect the result?

Answer 1

Standardization puts all the predictors on comparable scales, so that, for example, it wouldn't matter in terms of regression coefficients if length were measured in millimeters or miles. Otherwise, the penalty parameter lambda would not equally affect all the predictors.

With standardize=TRUE, glmnet standardizes all predictors before it does its calculations, then reports the penalized coefficients in the original, unstandardized scales of the predictors. Standardize=FALSE allows you instead to do your own standardization; for example, you might not want to standardize a dichotomous predictor by its mean and standard deviation, the default. But then you must take responsibility both for standardizing properly and for expressing coefficients in the original scales of the variables.

A toy example shows why standardized and non-standardized calculations will produce different coefficients with penalization. Say that you had a set of data in which exactly $y=x$, with $y$ in grams and $x$ in mm, with the empirical distribution of $x$ found to be $N$(0,1 mm). Then the coefficient that provides a perfect fit would be 1 gram/mm, and the penalty term in ridge regression at that perfect fit would be equal to $\lambda$. The final penalized coefficient would trade off the perfect fit against that penalty.

If you instead measured $x$ in centimeters but standardize $x$, you get the same tradeoff of fit against penalty as above. If you don't standardize in this case then you can still get a perfect fit to the data with a coefficient of 10 (now gram/cm), but the penalty in ridge regression to tradeoff against that is now $100\lambda$. No reason to suspect that the tradeoffs and the final choice of coefficients would be the same in the 2 cases. In your example, I suspect that differences between the empirical distributions and the normal distributions from which you sampled led to analogous, although less dramatic, differences in the penalized coefficients.