5
$\begingroup$

Suppose we have data $ (x_{1}, y_{1})\ldots (x_{N}, y_{N})$. The loss function of ridge regression is $$ \sum_i^N{(y_i - x^T_i\mathbf{\beta})^2} + \lambda \sum_j^p{\beta^2_j} $$

Notice that $ \sum_i^N{(y_i - x^T_i\mathbf{\beta})^2}$ term is related to number of all $(x_{i}, y_{i})$ pairs, increasing with length of $y$ vector. However, $\sum_j{\beta^2_j}$ is not.

How about $\lambda$ ?. Should it be related to $N$? It seems that if we keep $\lambda$ fixed, the penalty will be weaker with more data points. Am I wrong?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ That's a feature, not a bug! It means that increasing the sample size gets you closer and closer to the true $\beta$, assuming model correctness etc. You don't permanently penalize yourself away from the near-infinite $N$ solution, speaking very loosely. (You could have $\lambda \to \infty$ more slowly than $N$, but you'd have poorer rate-of-convergence to the true $\beta$ properties.) $\endgroup$
    – jbowman
    Commented Aug 10, 2015 at 3:47

1 Answer 1

Reset to default
4
$\begingroup$

There are many methods for selecting the regularization parameter ridge regression (also known as Tikhonov regularization) including the L-curve method, the discrepancy principle, and generalized cross validation. Assuming that the least squares problem is ill-conditioned so as to require regularization, all of the commonly used methods that I've just mentioned produce a value of $\lambda$ that depends not just on the number of data points, but also on the particular values of $y$. Using any of these methods, the value of $\lambda$ that is selected will typically increase with the number of data points.

In machine learning applications, it's typical to adjust $\lambda$ so that that the prediction error over the validation set is minimized. Here, the choice of $\lambda$ depends on both the training data and the validation data. As the size of the training set grows, it's typical for the optimal $\lambda$ to grow.

If you think of ridge regression as finding the Bayesian maximum a posteriori probability (MAP) solution beginning with a multivariate normal prior for the parameters $\beta$ and assuming a multivariate normal likelihood, then you get the ridge regression least squares problem to estimate the MAP solution. In this framework, since $\lambda$ is associated with the prior covariance it should not change as you add more data points.

Improve this answer
$\endgroup$
1
  • $\begingroup$ "As the size of the training set grows, it's typical for the optimal λ to grow." Is it safe to save it growly linearly? $\endgroup$
    – outsider
    Commented Oct 15, 2020 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.