From Elements of Statistical Learning page 64,

$loss = \sum (y- X\beta)^2 + \lambda \sum \beta^2$

The formula is not invariant to amount of data. If the data point doubles, $\lambda$ should be doubled?


1 Answer 1


Why should it be invariant to the amount of data? You are fitting model to a particular dataset, that has fixed size, so dividing by sample size does not change anything about the optimization, since it is a constant. This kind of constants are usually dropped for optimization.

As about choice of the regularizing parameter $\lambda$, it depends on many factors and there is no clear rules for choosing it, so it does not matter that much. In most cases you would need to find appropriate value of $\lambda$ by trial and error (a.k.a. hyperparameter tuning).

  • $\begingroup$ When you find $\lambda$ with 5 fold CV, and then fit the full data set, the amount of data increased by 25% $\endgroup$ Sep 26, 2019 at 1:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.