I want to do prediction using a ridge regression fit using the glmnet
package. There are 2 options of the standardize
parameter; 1) standardize=T
(which is the default) and 2) standardize=F
. I was thinking standardization of the data prior to regression is just for interpretation of the coefficients and it would not affect the prediction result. However, that's not what I see using the below example code (pred1
and pred2
are different from each other).
varsize = 50
samsize = 20
lambda = 1
set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps
trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]
library(glmnet)
pred1 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0, standardize=F), newx=testX)
print(cbind(pred1, pred2))
And the print result is:
[1,] 4.0063487 3.8358423
[2,] -0.2935879 -0.5569798
[3,] 0.8406129 0.3931324
[4,] -3.0215173 -1.7771050
[5,] -2.0329757 -4.1598014
[6,] 0.1703848 -0.3710678
[7,] -3.3436552 -3.2583682
[8,] -3.3409509 -2.6274531
[9,] -3.2172309 -4.0190585
[10,] -0.6965232 -0.3061931
However, with a linear regression, I would get the same result for pred1
and pred2
. Here is the example where just the $\lambda$ value is set to 0 in glmnet
call (so fitting a linear regression model rather than a ridge regression model):
varsize = 50
samsize = 20
set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps
trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]
library(glmnet)
pred1 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0, standardize=F), newx=testX)
print(cbind(pred1, pred2))
And here is the result:
[1,] -0.9765420 -0.9765420
[2,] 6.7416965 6.7416965
[3,] 3.6099447 3.6099447
[4,] -8.9710734 -8.9710734
[5,] 4.0057033 4.0057033
[6,] 6.9094331 6.9094331
[7,] -10.6853907 -10.6853907
[8,] -17.1785219 -17.1785219
[9,] -5.6884246 -5.6884246
[10,] -0.6812534 -0.6812534
Why is this behavior different in ridge regression? Why does standardization affect the result?
scale()
) during prediction. In general, I believe it is best to perform standardization externally and disable the default glmnet standardization. $\endgroup$