3
$\begingroup$

I have X possible predictors for response Y. In my case X >> Y.

I have noticed in my runs of cv.glmnet (leave-on-out and all other params default) that if I try to predict using lambda.min that it simply returns the mean value of Y. If I run the prediction with choices of lambda < lambda.min, it gives actual predictions - which have a lower error than using the mean value of Y.

I'm not sure what's going on here. It's as if the code is defaulting to a dummy predictor (the mean response) for some reason. It appears that this behavior is a function of the size of X.

Here's a simple example:

x=replicate(100,rnorm(10))

y=replicate(1,rnorm(10))

cvfit=cv.glmnet(x,y,nfolds=10)

ypred1=predict(cvfit,newx=x,s="lambda.min")

(in a case I just ran, this gives a cvfit$lambda.min = 0.8453387 and all entries in ypred1 are the mean value of y. So, let's choose a different lambda)

ypred2=predict(cvfit,newx=x,s=0.1)

mse1=mean((ypred1-y)^2) = 1.20

mse2=mean((ypred2-y)^2) = 0.03

I understand that "newx=x" doesn't make sense for any real work, but I don't understand why it returns the predictions it does.

$\endgroup$
  • 3
    $\begingroup$ Can you provide a reproducible example? Also, I believe it's actually generally recommended to use lambda.1se over lambda.min $\endgroup$ – David Marx Apr 12 '14 at 18:29
  • $\begingroup$ Thanks for the comment. I'm not sure how to do this. In my case, the size of X (number of potential predictors) is > 500. Also, lambda.1se = lambda.min when I see this happen. $\endgroup$ – PickledZebra Apr 12 '14 at 18:53
  • 1
    $\begingroup$ I strongly suspect there's something wonky with your code or the data you are feeding your model. lambda.1se should be larger than lambda.min. Without access to your code and/or data that reproduces the issue, I don't really know what to tell you. $\endgroup$ – David Marx Apr 12 '14 at 19:19
  • $\begingroup$ I've tried to create an example and edited the original post. $\endgroup$ – PickledZebra Apr 12 '14 at 19:32
  • 1
    $\begingroup$ Try to cook up a more meaningful testcase than x,y using rnorm(), even if you did set.seed(). You wouldn't expect y ~ x. Pick one of the builtin R datasets. $\endgroup$ – smci Feb 10 '17 at 14:50
6
$\begingroup$

Here, glmnet is working as intended! In your example, there is no relationship between $x$ and $y$ (both were independently generated). So the ``correct'' thing to do is to just always predict $\hat{y} = \bar{y}.$ Any method that isn't doing that is overfitting the test set.

$\endgroup$
  • $\begingroup$ I understand that. But, in the case I show, glmnet did find a fit. It can overfit even if the two sets were independently generated. I don't understand how the algorithm determines that the model it generated was overfit and decided to do the "correct" thing. $\endgroup$ – PickledZebra Apr 12 '14 at 20:09
2
$\begingroup$

This is simply a case of how glmnet generates a default sequence of lambdas. It calculates the maximum lambda in its grid search based on a computation from your data, it is essentially the minimal lambda that sets all the coefficients in your model equal to zero.

The details are in this paper, but the rub is that:

$$ N \alpha \lambda_{max} = \max_l \| \langle x_l, y \rangle \| $$

Since the correct answer in your case is to zero out all (non-intercept) coefficients, I would guess it quite likely that cv.glment is returning this $\lambda_{max}$ as optimal.

$\endgroup$
  • $\begingroup$ This is missing the important part: "Never rely on the "default" lambda sequence, always supply your own one" $\endgroup$ – smci Feb 24 '18 at 5:48
1
$\begingroup$

lambda.1se == lambda.min

"All entries in ypred1 are the mean value of y"

Both of these tell you that your coefficients got zeroed out. (You should always inspect the coefficients with coef(cvfit, s='lambda.1se')) Either lambda was too small, or your x variables weren't normalized. Or they simply aren't predictive of y.

Never rely on the "default" lambda sequence, always supply your own one:

cv.glmnet(..., lambda=10^(seq(m,-n,0.2)))

So rerun with that/ Also, show us your deviance plot: plot.cv.glmnet(cvfit, sign.lambda=-1) There's supposed to be a knee in it. Yours apparently doesn't.

If you double-checked all the above, then your x-variables simply aren't predictive of y. (unless you made a gross error and you scrambled x,y)

$\endgroup$
  • $\begingroup$ Could you extend this statement "Never rely on the "default" lambda sequence, always supply your own one" ? $\endgroup$ – rook1996 Jun 11 '18 at 10:44
  • $\begingroup$ @rook1996: I already do, I show a user-defined lambda sequence lambda=10^(seq(m,-n,0.2)) $\endgroup$ – smci Jun 11 '18 at 22:03
  • 1
    $\begingroup$ No I mean, why should I supply my own Lambda sequence. What is wrong with the calculation per default Lambda ? Do you have some sources where this argument is stated ? And which values should I choose for m and n $\endgroup$ – rook1996 Jun 11 '18 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.