# Ridge Regression - what is the relationship between $\lambda$ and the data's noise?

Currently reviewing some problems in my ML class, and I came across this problem:

You estimate a ridge regression model with some data taken from your robot, and find (using cross validation) and optimal ridge penalty $$\lambda_1$$. You then buy a new sensor which has noise with $$1/4$$ the variance (half the standard deviation) as before. Using the same number of observations as before you collect new data, and find a new optimal ridge penalty $$\lambda_2$$.

Which of the following will be closest to true?

1. $$\lambda_1 / \lambda_2 = 1/4$$
2. $$\lambda_1 / \lambda_2 = 1/2$$
3. $$\lambda_1 / \lambda_2 = 1$$
4. $$\lambda_1 / \lambda_2 = 2$$
5. $$\lambda_1 / \lambda_2 = 4$$

The solution key indicates that the answer is $$\lambda_1 / \lambda_2 = 4$$, but does not provide an explanation.

Here is my attempt at solving the problem:

From an assumption that $$y_i = \mathbf{x}_i^\top \mathbf{w}^* + \epsilon_i$$ where each error epsilon is drawn from a gaussian distribution with 0 mean and variance $$\sigma^2$$ (i.e. $$\epsilon_i \sim N(0, \sigma^2)$$), we have that ridge regression is

$$\widehat{\mathbf{w}}_\text{ridge} = \arg\underset{\mathbf{w}}{\min} \sum_{i=1}^{n}(y_i - \mathbf{x}_i^\top \mathbf{w})^2 + \lambda \|\mathbf{w} \|_2^2$$

The closed form solution for $$\widehat{\mathbf{w}}_\text{ridge}$$ is $$\widehat{\mathbf{w}}_\text{ridge} = (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y}$$

When calculating the bias-variance tradeoff, we get \begin{aligned} \widehat{\mathbf{w}}_\text{ridge} &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top(\mathbf{X}\mathbf{w}^* + \mathbf{\epsilon}) && \mathbf{y} = \mathbf{X}\mathbf{w}^* + \mathbf{\epsilon} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}(\mathbf{X}^\top\mathbf{X}\mathbf{w}^* + \mathbf{X}^\top\mathbf{\epsilon}) \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}(\mathbf{X}^\top\mathbf{X}\mathbf{w}^* + \mathbf{X}^\top\mathbf{\epsilon} + \lambda\mathbf{I{w^*}} - \lambda\mathbf{I{w^*}}) &&\text{adding zero} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}((\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})\mathbf{w}^* - \lambda\mathbf{I{w^*}} + \mathbf{X}^\top\mathbf{\epsilon}) &&\text{rearranging terms} \\ &= \mathbf{w}^* - (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\lambda\mathbf{w}^* + (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{\epsilon} &&\text{distribute} \end{aligned}

If our sensor has noise with $$1/4$$ the variance, then our new error $${\epsilon_\text{new}}_i \sim N(0, \frac{\sigma^2}{4}) = \frac{{\epsilon_\text{old}}_i}{2}$$. From there, we can conclude that $$\mathbf{\epsilon}_\text{new} = \frac{\mathbf{\epsilon}_\text{old}}{2}$$.

Does this somehow translate into the variance term $$(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{\epsilon}$$? I'm struggling to see how $$\lambda_1/\lambda_2 = 4$$, as this would imply that $$\lambda_\text{new} = \frac{\lambda_1}{4}$$. Is there some form of correlation between the distribution that $$\epsilon$$ is drawn from and $$\lambda$$?

Would appreciate any suggestions to approach this problem - please let me know if my logic is also off anywhere. Thanks in advance!

• The question implicitly assumes all the variation in the response is sensor "noise."
– whuber
Nov 8, 2022 at 13:30

Maybe this problem is motivated by a connection to Bayesian Ridge Regression with assumptions $$\mathbf y\,|\,\mathbf w, \sigma^2 \sim \mathcal N(\mathbf X \mathbf w, \sigma^2\mathbf I)$$ and $$\mathbf w \,|\, \tau^2 \sim \mathcal N(\mathbf 0, \tau^2\mathbf I)$$. In this setting, maximizing the posterior density $$p(\mathbf w\,|\, \mathbf y, \sigma^2, \tau^2)$$ w.r.t. $$\mathbf w$$ for fixed $$\sigma^2, \tau^2$$ is equivalent to minimizing the Ridge-penalized least squares criterion for a fixed $$\lambda \equiv \sigma^2/\tau^2$$ since \begin{align} p(\mathbf w\,|\, \mathbf y, \sigma^2, \tau^2)& \propto p(\mathbf y\,|\,\mathbf w, \sigma^2) \cdot p(\mathbf w \,|\, \tau^2)\\ & \propto \exp\left(-\frac{1}{2\sigma^2}\left\|\mathbf y - \mathbf X \mathbf w\right\|_2^2\right) \cdot \exp\left(-\frac{1}{2\tau^2}\left\|\mathbf w\right\|_2^2\right) \\ &=\exp\left(-\frac{1}{2\sigma^2}\left(\left\|\mathbf y - \mathbf X \mathbf w\right\|_2^2 + \frac{\sigma^2}{\tau^2}\left\|\mathbf w\right\|_2^2\right)\right). \end{align} However, as pointed out in Sextus Empiricus' answer, this does not imply that when $$\sigma^2$$ changes the optimal (w.r.t. cross-validation) $$\lambda$$ has to change by the same factor.
The naive approach will be to use the cost function $$\sum_{i=1}^n (\hat{y} - y)^2 + \lambda \sum_{i=1}^p \beta_i^2$$ and consider that when the first term decreases by 4 because of the better sensor, then the second term can be decreased by 4 as well.
However, possibly it is not so trivial. When we decrease the penalty $$\lambda$$ by a factor $$4$$, then this will have a sort of feedback effect on the first term which initially decreased by $$4$$ but now will be larger again because the predictions $$\hat{\beta}$$ have changed.
What we can say is that the answer will be between two values $$1< \lambda_1/\lambda_2 < 4$$ but whether it is closer to $$1$$, $$2$$, or $$4$$ depends on the specific problem.