3
$\begingroup$

I am running a glmnet fit with 1200000 samples.

According to the glmnet doc, $\lambda$ value is the coefficient controlling how much the regularization term contributes to the total loss function.

For my case, the best $\lambda$ is the fifth in the lambda list, which is 1.037156e-05. I think this value is suspiciously small. Because the mean square error is 7.37544e-05. The total loss from $||y-y_{hat}||_{2}$ is around 88. However, the sum of L1 norm $||\beta||_{1}$ is roughly 0.02, if you further scale it down with that small $\lambda$, the regularization loss is disproportionately smaller than the square error loss.

This is where my confusion comes from. If the regularization term's contribution is so small, how can it regularize anything?

I simply cannot make sense out of this. Any chance the real loss function that glmnet uses under the hood is different from what the authors presented in the web page? or there is something wrong in my modeling/calculation?

Just for the sanity:

cvfit = cv.glmnet(
    x = X,
    y = Y,
    alpha = 1,
    family = "gaussian",
    intercept = FALSE,
    nlambda = 20,
    standardize = FALSE,
    type.measure = 'mae',
    parallel = TRUE
)
$\endgroup$
4
  • $\begingroup$ Try fitting the model on a small subset (say $n = 100$) and observe the optimal $\lambda$, then decrease the sample size further and further... can you see what's happening? $\endgroup$ Sep 24, 2019 at 9:23
  • $\begingroup$ @FransRodenburg I tried with 100 samples, the new lambda is around 0.0018, is this expected? $\endgroup$
    – eight3
    Sep 24, 2019 at 9:37
  • $\begingroup$ Slightly off-topic, but why have you set the intercept to FALSE? $\endgroup$ Sep 24, 2019 at 12:19
  • $\begingroup$ there has been a long lasting war in my team about this option. The best selling explanation is: glmnet does not scale the interception; which give it an unfair advantage over the other predictors. Also, we want the prediction to be 0 when all the predictors are zeros. $\endgroup$
    – eight3
    Sep 24, 2019 at 13:17

1 Answer 1

2
$\begingroup$

Think of regularization as a way to penalize for model complexity. If the model complexity is high when compared to the sample size, it is very prone to overfitting. Choosing the optimal $\lambda$ through cross-validation can then prevent this from happening.

However, if like in your case, you have an extremely large sample size, then most models will have nowhere near the model complexity to warrant a substantial amount of shrinkage. This is why the optimal $\lambda$ through CV will be very close to $0$.

You could try a more complex model, e.g. by including more variables, interactions, or by base expansion. Or you might simply not need regularization.

$\endgroup$
2
  • $\begingroup$ nevertheless, glmnet removed 2/3 of the predictors ,i.e., set their betas to be 0. That means regularization still works well in my original case. $\endgroup$
    – eight3
    Sep 24, 2019 at 9:41
  • $\begingroup$ If that is desirable, then by all means keep using the regularized model and don't worry about the particular value of $\lambda$. You could consider re-estimating on the remaining predictors, such as is done in the relaxed LASSO. $\endgroup$ Sep 24, 2019 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.