3
$\begingroup$

Is it true that:

Suppose you perform linear regression with $L_2$ regularization and use cross-validation to select the value of the regularization parameter $λ$ on two datasets drawn from the same distribution : $D_1$ of 500 examples and $D_2$ of 50,000 examples. The value of lambda found by cross-validation will likely be higher on $D_2$ than on $D_1$.

$\endgroup$
2
  • $\begingroup$ Can you specify what you mean by examples? Do you mean folds? There are several cross-validation methods available, but I guess that more times you perform the folding, the more accurate will be the lambda term supposed you have sufficient size of the data set. This means it might be higher for some, and lower for some other variables. $\endgroup$ Oct 29, 2019 at 6:48
  • $\begingroup$ What type of standardization, if any, is applied to X? $\endgroup$ Jun 1, 2020 at 9:37

1 Answer 1

5
$\begingroup$

Regularization constrains the parameter space of a model that would otherwise overfit the sample. The optimal amount of regularization depends on the model complexity relative to the sample size ($n$). Namely, the smaller the sample and/or the complexer the model, the more prone the model is to overfitting. Cross-validation should result in a model that is regularized to the extent that it no longer overfits. Hence, the optimal value $\lambda_{\text{CV}}$ grows roughly with the ratio $\frac{p}{n}$, where $p$ is the number of parameters.

Because of this, if the same model is fit on on $D_1$ ($n=500$) and $D_2$ ($n=50,000$), then the optimal value of $\lambda$ will almost certainly be lower for $D_2$ than for $D_1$, because $p$ is constant, and $\frac{p}{500} > \frac{p}{50,000}$.

Put differently, the relative model complexity is lower when you have more observations, so $D_2$ requires less regularization (if any) to combat overfitting.

$\endgroup$
4
  • 2
    $\begingroup$ Do you have a citation for the claim that $\lambda_{CV}=O\left(\frac{p}{n}\right)$? $\endgroup$
    – cfp
    Nov 27, 2019 at 9:57
  • $\begingroup$ It'd also be great to have your thoughts on my related question here: stats.stackexchange.com/questions/438173/… . $\endgroup$
    – cfp
    Nov 27, 2019 at 10:39
  • $\begingroup$ @cfp My wording may have been confusing, I did not mean to say that the relationship is linear. I have made a small edit to this answer and posted an answer to your linked question. $\endgroup$ Nov 27, 2019 at 16:01
  • $\begingroup$ I guess I should have written $O_p$ (i.e. order of, in probability). I certainly didn't think you meant that it was a deterministic function! I'll comment further on your question answer. $\endgroup$
    – cfp
    Nov 27, 2019 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.