# How does size of training set affect the regularization parameter found by cross validation?

Is it true that:

Suppose you perform linear regression with $$L_2$$ regularization and use cross-validation to select the value of the regularization parameter $$λ$$ on two datasets drawn from the same distribution : $$D_1$$ of 500 examples and $$D_2$$ of 50,000 examples. The value of lambda found by cross-validation will likely be higher on $$D_2$$ than on $$D_1$$.

• Can you specify what you mean by examples? Do you mean folds? There are several cross-validation methods available, but I guess that more times you perform the folding, the more accurate will be the lambda term supposed you have sufficient size of the data set. This means it might be higher for some, and lower for some other variables. – Vlatko Galic Oct 29 '19 at 6:48
• What type of standardization, if any, is applied to X? – Johan Larsson Jun 1 at 9:37

Regularization constrains the parameter space of a model that would otherwise overfit the sample. The optimal amount of regularization depends on the model complexity relative to the sample size ($$n$$). Namely, the smaller the sample and/or the complexer the model, the more prone the model is to overfitting. Cross-validation should result in a model that is regularized to the extent that it no longer overfits. Hence, the optimal value $$\lambda_{\text{CV}}$$ grows roughly with the ratio $$\frac{p}{n}$$, where $$p$$ is the number of parameters.
Because of this, if the same model is fit on on $$D_1$$ ($$n=500$$) and $$D_2$$ ($$n=50,000$$), then the optimal value of $$\lambda$$ will almost certainly be lower for $$D_2$$ than for $$D_1$$, because $$p$$ is constant, and $$\frac{p}{500} > \frac{p}{50,000}$$.
Put differently, the relative model complexity is lower when you have more observations, so $$D_2$$ requires less regularization (if any) to combat overfitting.
• Do you have a citation for the claim that $\lambda_{CV}=O\left(\frac{p}{n}\right)$? – cfp Nov 27 '19 at 9:57
• I guess I should have written $O_p$ (i.e. order of, in probability). I certainly didn't think you meant that it was a deterministic function! I'll comment further on your question answer. – cfp Nov 27 '19 at 16:11