3
$\begingroup$

I want to do prediction using a ridge regression fit using the glmnet package. There are 2 options of the standardize parameter; 1) standardize=T (which is the default) and 2) standardize=F. I was thinking standardization of the data prior to regression is just for interpretation of the coefficients and it would not affect the prediction result. However, that's not what I see using the below example code (pred1 and pred2 are different from each other).

varsize = 50
samsize = 20
lambda = 1

set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps

trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]

library(glmnet)

pred1 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=lambda, alpha = 0, standardize=F), newx=testX)

print(cbind(pred1, pred2))

And the print result is:

 [1,]  4.0063487  3.8358423
 [2,] -0.2935879 -0.5569798
 [3,]  0.8406129  0.3931324
 [4,] -3.0215173 -1.7771050
 [5,] -2.0329757 -4.1598014
 [6,]  0.1703848 -0.3710678
 [7,] -3.3436552 -3.2583682
 [8,] -3.3409509 -2.6274531
 [9,] -3.2172309 -4.0190585
[10,] -0.6965232 -0.3061931

However, with a linear regression, I would get the same result for pred1 and pred2. Here is the example where just the $\lambda$ value is set to 0 in glmnet call (so fitting a linear regression model rather than a ridge regression model):

varsize = 50
samsize = 20

set.seed(333)
X = matrix(rnorm(samsize*varsize), ncol=varsize)
set.seed(343)
w = matrix(rnorm(varsize), ncol=1)
set.seed(353)
eps = matrix(rnorm(samsize), ncol=1)
y = X %*% w + eps

trainsamp = 1:10
testsamp = 11:20
trainX = X[trainsamp,]
trainy = y[trainsamp]
testX = X[testsamp,]

library(glmnet)

pred1 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0), newx=testX)
pred2 = predict(glmnet(trainX, trainy, lambda=0, alpha = 0, standardize=F), newx=testX)

print(cbind(pred1, pred2))

And here is the result:

 [1,]  -0.9765420  -0.9765420
 [2,]   6.7416965   6.7416965
 [3,]   3.6099447   3.6099447
 [4,]  -8.9710734  -8.9710734
 [5,]   4.0057033   4.0057033
 [6,]   6.9094331   6.9094331
 [7,] -10.6853907 -10.6853907
 [8,] -17.1785219 -17.1785219
 [9,]  -5.6884246  -5.6884246
[10,]  -0.6812534  -0.6812534

Why is this behavior different in ridge regression? Why does standardization affect the result?

$\endgroup$
5
  • 1
    $\begingroup$ The documentation explicitly says not to set lambda to a single constant. $\endgroup$ Feb 8, 2017 at 2:13
  • $\begingroup$ This was just for example purposes. I choose lambda by cross validation in the actual code. $\endgroup$
    – user5054
    Feb 8, 2017 at 6:36
  • 1
    $\begingroup$ Sure, but when the documentation explicitly says not to do something, you shouldn't do it even in an example. If you are misusing the library, there is no reason to trust the results. $\endgroup$ Feb 8, 2017 at 16:36
  • $\begingroup$ That's not the point here, I guess. The library suggests not to supply a single lambda value because the user is never able to know which lambda value would give a reasonable or meaningful result, so checking for a series of lambda values and choosing the one with the most meaningful result (smallest error for example) is what one should do. But here, I am not interested in the result itself at all, I am just interested in comparison of it with the result when a parameter value is set to a different option. So, a single lambda value would be completely fine here. $\endgroup$
    – user5054
    Feb 8, 2017 at 22:19
  • $\begingroup$ You will also observe drastic differences when standardizing the test set (eg. using scale()) during prediction. In general, I believe it is best to perform standardization externally and disable the default glmnet standardization. $\endgroup$
    – Megatron
    Oct 30, 2020 at 17:12

1 Answer 1

6
$\begingroup$

Standardization puts all the predictors on comparable scales, so that, for example, it wouldn't matter in terms of regression coefficients if length were measured in millimeters or miles. Otherwise, the penalty parameter lambda would not equally affect all the predictors.

With standardize=TRUE, glmnet standardizes all predictors before it does its calculations, then reports the penalized coefficients in the original, unstandardized scales of the predictors. Standardize=FALSE allows you instead to do your own standardization; for example, you might not want to standardize a dichotomous predictor by its mean and standard deviation, the default. But then you must take responsibility both for standardizing properly and for expressing coefficients in the original scales of the variables.

A toy example shows why standardized and non-standardized calculations will produce different coefficients with penalization. Say that you had a set of data in which exactly $y=x$, with $y$ in grams and $x$ in mm, with the empirical distribution of $x$ found to be $N$(0,1 mm). Then the coefficient that provides a perfect fit would be 1 gram/mm, and the penalty term in ridge regression at that perfect fit would be equal to $\lambda$. The final penalized coefficient would trade off the perfect fit against that penalty.

If you instead measured $x$ in centimeters but standardize $x$, you get the same tradeoff of fit against penalty as above. If you don't standardize in this case then you can still get a perfect fit to the data with a coefficient of 10 (now gram/cm), but the penalty in ridge regression to tradeoff against that is now $100\lambda$. No reason to suspect that the tradeoffs and the final choice of coefficients would be the same in the 2 cases. In your example, I suspect that differences between the empirical distributions and the normal distributions from which you sampled led to analogous, although less dramatic, differences in the penalized coefficients.

$\endgroup$
2
  • $\begingroup$ Thanks for the reply. I in fact know what standardization is and when to use it, but I don't understand why the glmnet prediction is affected by standardization. $\endgroup$
    – user5054
    Feb 8, 2017 at 6:39
  • $\begingroup$ @user5054 added an example that might help. $\endgroup$
    – EdM
    Feb 8, 2017 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.