# Should the lambda of ridge regression be related to number of data points?

Suppose we have data $(x_{1}, y_{1})\ldots (x_{N}, y_{N})$. The loss function of ridge regression is $$\sum_i^N{(y_i - x^T_i\mathbf{\beta})^2} + \lambda \sum_j^p{\beta^2_j}$$

Notice that $\sum_i^N{(y_i - x^T_i\mathbf{\beta})^2}$ term is related to number of all $(x_{i}, y_{i})$ pairs, increasing with length of $y$ vector. However, $\sum_j{\beta^2_j}$ is not.

How about $\lambda$ ?. Should it be related to $N$? It seems that if we keep $\lambda$ fixed, the penalty will be weaker with more data points. Am I wrong?

• That's a feature, not a bug! It means that increasing the sample size gets you closer and closer to the true $\beta$, assuming model correctness etc. You don't permanently penalize yourself away from the near-infinite $N$ solution, speaking very loosely. (You could have $\lambda \to \infty$ more slowly than $N$, but you'd have poorer rate-of-convergence to the true $\beta$ properties.) Aug 10, 2015 at 3:47

There are many methods for selecting the regularization parameter ridge regression (also known as Tikhonov regularization) including the L-curve method, the discrepancy principle, and generalized cross validation. Assuming that the least squares problem is ill-conditioned so as to require regularization, all of the commonly used methods that I've just mentioned produce a value of $\lambda$ that depends not just on the number of data points, but also on the particular values of $y$. Using any of these methods, the value of $\lambda$ that is selected will typically increase with the number of data points.
In machine learning applications, it's typical to adjust $\lambda$ so that that the prediction error over the validation set is minimized. Here, the choice of $\lambda$ depends on both the training data and the validation data. As the size of the training set grows, it's typical for the optimal $\lambda$ to grow.
If you think of ridge regression as finding the Bayesian maximum a posteriori probability (MAP) solution beginning with a multivariate normal prior for the parameters $\beta$ and assuming a multivariate normal likelihood, then you get the ridge regression least squares problem to estimate the MAP solution. In this framework, since $\lambda$ is associated with the prior covariance it should not change as you add more data points.