From Elements of Statistical Learning page 64,
$loss = \sum (y- X\beta)^2 + \lambda \sum \beta^2$
The formula is not invariant to amount of data. If the data point doubles, $\lambda$ should be doubled?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Why should it be invariant to the amount of data? You are fitting model to a particular dataset, that has fixed size, so dividing by sample size does not change anything about the optimization, since it is a constant. This kind of constants are usually dropped for optimization.
As about choice of the regularizing parameter $\lambda$, it depends on many factors and there is no clear rules for choosing it, so it does not matter that much. In most cases you would need to find appropriate value of $\lambda$ by trial and error (a.k.a. hyperparameter tuning).
-
$\begingroup$ When you find $\lambda$ with 5 fold CV, and then fit the full data set, the amount of data increased by 25% $\endgroup$ Commented Sep 26, 2019 at 1:23