Is it true that:

Suppose you perform linear regression with $L_2$ regularization and use cross-validation to select the value of the regularization parameter $λ$ on two datasets drawn from the same distribution : $D_1$ of 500 examples and $D_2$ of 50,000 examples. The value of lambda found by cross-validation will likely be higher on $D_2$ than on $D_1$.

  • $\begingroup$ Can you specify what you mean by examples? Do you mean folds? There are several cross-validation methods available, but I guess that more times you perform the folding, the more accurate will be the lambda term supposed you have sufficient size of the data set. This means it might be higher for some, and lower for some other variables. $\endgroup$ – Vlatko Galic Oct 29 '19 at 6:48
  • $\begingroup$ What type of standardization, if any, is applied to X? $\endgroup$ – Johan Larsson Jun 1 at 9:37

Regularization constrains the parameter space of a model that would otherwise overfit the sample. The optimal amount of regularization depends on the model complexity relative to the sample size ($n$). Namely, the smaller the sample and/or the complexer the model, the more prone the model is to overfitting. Cross-validation should result in a model that is regularized to the extent that it no longer overfits. Hence, the optimal value $\lambda_{\text{CV}}$ grows roughly with the ratio $\frac{p}{n}$, where $p$ is the number of parameters.

Because of this, if the same model is fit on on $D_1$ ($n=500$) and $D_2$ ($n=50,000$), then the optimal value of $\lambda$ will almost certainly be lower for $D_2$ than for $D_1$, because $p$ is constant, and $\frac{p}{500} > \frac{p}{50,000}$.

Put differently, the relative model complexity is lower when you have more observations, so $D_2$ requires less regularization (if any) to combat overfitting.

| cite | improve this answer | |
  • $\begingroup$ Do you have a citation for the claim that $\lambda_{CV}=O\left(\frac{p}{n}\right)$? $\endgroup$ – cfp Nov 27 '19 at 9:57
  • $\begingroup$ It'd also be great to have your thoughts on my related question here: stats.stackexchange.com/questions/438173/… . $\endgroup$ – cfp Nov 27 '19 at 10:39
  • $\begingroup$ @cfp My wording may have been confusing, I did not mean to say that the relationship is linear. I have made a small edit to this answer and posted an answer to your linked question. $\endgroup$ – Frans Rodenburg Nov 27 '19 at 16:01
  • $\begingroup$ I guess I should have written $O_p$ (i.e. order of, in probability). I certainly didn't think you meant that it was a deterministic function! I'll comment further on your question answer. $\endgroup$ – cfp Nov 27 '19 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.