4
$\begingroup$

I have X possible predictors for response Y. In my case X >> Y.

I have noticed in my runs of cv.glmnet (leave-on-out and all other params default) that if I try to predict using lambda.min that it simply returns the mean value of Y. If I run the prediction with choices of lambda < lambda.min, it gives actual predictions - which have a lower error than using the mean value of Y.

I'm not sure what's going on here. It's as if the code is defaulting to a dummy predictor (the mean response) for some reason. It appears that this behavior is a function of the size of X.

Here's a simple example:

x=replicate(100,rnorm(10))

y=replicate(1,rnorm(10))

cvfit=cv.glmnet(x,y,nfolds=10)

ypred1=predict(cvfit,newx=x,s="lambda.min")

(in a case I just ran, this gives a cvfit$lambda.min = 0.8453387 and all entries in ypred1 are the mean value of y. So, let's choose a different lambda)

ypred2=predict(cvfit,newx=x,s=0.1)

mse1=mean((ypred1-y)^2) = 1.20

mse2=mean((ypred2-y)^2) = 0.03

I understand that "newx=x" doesn't make sense for any real work, but I don't understand why it returns the predictions it does.

$\endgroup$
8
  • 3
    $\begingroup$ Can you provide a reproducible example? Also, I believe it's actually generally recommended to use lambda.1se over lambda.min $\endgroup$
    – David Marx
    Apr 12 '14 at 18:29
  • $\begingroup$ Thanks for the comment. I'm not sure how to do this. In my case, the size of X (number of potential predictors) is > 500. Also, lambda.1se = lambda.min when I see this happen. $\endgroup$ Apr 12 '14 at 18:53
  • 1
    $\begingroup$ I strongly suspect there's something wonky with your code or the data you are feeding your model. lambda.1se should be larger than lambda.min. Without access to your code and/or data that reproduces the issue, I don't really know what to tell you. $\endgroup$
    – David Marx
    Apr 12 '14 at 19:19
  • $\begingroup$ I've tried to create an example and edited the original post. $\endgroup$ Apr 12 '14 at 19:32
  • 1
    $\begingroup$ Try to cook up a more meaningful testcase than x,y using rnorm(), even if you did set.seed(). You wouldn't expect y ~ x. Pick one of the builtin R datasets. $\endgroup$
    – smci
    Feb 10 '17 at 14:50
8
$\begingroup$

Here, glmnet is working as intended! In your example, there is no relationship between $x$ and $y$ (both were independently generated). So the ``correct'' thing to do is to just always predict $\hat{y} = \bar{y}.$ Any method that isn't doing that is overfitting the test set.

$\endgroup$
1
  • $\begingroup$ I understand that. But, in the case I show, glmnet did find a fit. It can overfit even if the two sets were independently generated. I don't understand how the algorithm determines that the model it generated was overfit and decided to do the "correct" thing. $\endgroup$ Apr 12 '14 at 20:09
4
$\begingroup$

This is simply a case of how glmnet generates a default sequence of lambdas. It calculates the maximum lambda in its grid search based on a computation from your data, it is essentially the minimal lambda that sets all the coefficients in your model equal to zero.

The details are in this paper, but the rub is that:

$$ N \alpha \lambda_{max} = \max_l \| \langle x_l, y \rangle \| $$

Since the correct answer in your case is to zero out all (non-intercept) coefficients, I would guess it quite likely that cv.glment is returning this $\lambda_{max}$ as optimal.

$\endgroup$
1
  • $\begingroup$ This is missing the important part: "Never rely on the "default" lambda sequence, always supply your own one" $\endgroup$
    – smci
    Feb 24 '18 at 5:48
2
$\begingroup$

lambda.1se == lambda.min

"All entries in ypred1 are the mean value of y"

Both of these tell you that your coefficients got zeroed out. (You should always inspect the coefficients with coef(cvfit, s='lambda.1se')) Either lambda was too small, or your x variables weren't normalized. Or they simply aren't predictive of y.

Never rely on the "default" lambda sequence, always supply your own one:

cv.glmnet(..., lambda=10^(seq(m,-n,0.2)))

which uses a lambda geometrically from 10^m ... 10^(-n). You choose m, -n to be wide enough to cover the range that includes all possible optimal values of lambda.

So rerun with that, then pick the right lambda./ Also, show us your deviance plot: plot.cv.glmnet(cvfit, sign.lambda=-1) There's supposed to be a knee in it. Yours apparently doesn't.

If you double-checked all the above, then your x-variables simply aren't predictive of y. (unless you made a gross error and you scrambled x,y)

$\endgroup$
20
  • 2
    $\begingroup$ Could you extend this statement "Never rely on the "default" lambda sequence, always supply your own one" ? $\endgroup$
    – rook1996
    Jun 11 '18 at 10:44
  • 4
    $\begingroup$ No I mean, why should I supply my own Lambda sequence. What is wrong with the calculation per default Lambda ? Do you have some sources where this argument is stated ? And which values should I choose for m and n $\endgroup$
    – rook1996
    Jun 11 '18 at 22:17
  • 3
    $\begingroup$ @smci in your comment you state "Never rely on the "default" lambda sequence, always supply your own one", where and why is there evidence that supports this? glmnet documentation doesn't necessarily state that so I was wondering if there was documentation elsewhere. $\endgroup$
    – AW27
    Feb 17 '21 at 11:11
  • 3
    $\begingroup$ The main issue I think everyone is having with the "just because" selection of lambda in your example is that you can't guarantee that your sequence with whatever m and n will cover the necessary range of possible lambda values. In literature, however, maximum value is well documented, as for the minimum value, don't forget that glmnet also implements various heuristic rules, e.g., Tibshirani et. al., among other. The idea is to stop before reaching the minimum value $\endgroup$
    – runr
    Jun 16 '21 at 21:46
  • 2
    $\begingroup$ @EmmanuelGoldstein You might consider asking a question about the suitability of the glmnet defaults. There's no need to mention this answer or users in this thread -- just state in specific but neutral terms that you had someone tell you about this deficiency and you want to know if there is an academic reference that discusses it in detail, a mathematical demonstration, etc. $\endgroup$
    – Sycorax
    Jun 17 '21 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.