Ridge Regression - what is the relationship between $\lambda$ and the data's noise?

Question

Currently reviewing some problems in my ML class, and I came across this problem:

You estimate a ridge regression model with some data taken from your robot, and find (using cross validation) and optimal ridge penalty $\lambda_1$. You then buy a new sensor which has noise with $1/4$ the variance (half the standard deviation) as before. Using the same number of observations as before you collect new data, and find a new optimal ridge penalty $\lambda_2$.

Which of the following will be closest to true?

1. $\lambda_1 / \lambda_2 = 1/4$
2. $\lambda_1 / \lambda_2 = 1/2$
3. $\lambda_1 / \lambda_2 = 1$
4. $\lambda_1 / \lambda_2 = 2$
5. $\lambda_1 / \lambda_2 = 4$

The solution key indicates that the answer is $\lambda_1 / \lambda_2 = 4$, but does not provide an explanation.

Here is my attempt at solving the problem:

From an assumption that $y_i = \mathbf{x}_i^\top \mathbf{w}^* + \epsilon_i$ where each error epsilon is drawn from a gaussian distribution with 0 mean and variance $\sigma^2$ (i.e. $\epsilon_i \sim N(0, \sigma^2)$), we have that ridge regression is

$$ \widehat{\mathbf{w}}_\text{ridge} = \arg\underset{\mathbf{w}}{\min} \sum_{i=1}^{n}(y_i - \mathbf{x}_i^\top \mathbf{w})^2 + \lambda \|\mathbf{w} \|_2^2 $$

The closed form solution for $\widehat{\mathbf{w}}_\text{ridge}$ is $$ \widehat{\mathbf{w}}_\text{ridge} = (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y} $$

When calculating the bias-variance tradeoff, we get $$ \begin{aligned} \widehat{\mathbf{w}}_\text{ridge} &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top(\mathbf{X}\mathbf{w}^* + \mathbf{\epsilon}) && \mathbf{y} = \mathbf{X}\mathbf{w}^* + \mathbf{\epsilon} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}(\mathbf{X}^\top\mathbf{X}\mathbf{w}^* + \mathbf{X}^\top\mathbf{\epsilon}) \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}(\mathbf{X}^\top\mathbf{X}\mathbf{w}^* + \mathbf{X}^\top\mathbf{\epsilon} + \lambda\mathbf{I{w^*}} - \lambda\mathbf{I{w^*}}) &&\text{adding zero} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}((\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})\mathbf{w}^* - \lambda\mathbf{I{w^*}} + \mathbf{X}^\top\mathbf{\epsilon}) &&\text{rearranging terms} \\ &= \mathbf{w}^* - (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\lambda\mathbf{w}^* + (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{\epsilon} &&\text{distribute} \end{aligned} $$

If our sensor has noise with $1/4$ the variance, then our new error ${\epsilon_\text{new}}_i \sim N(0, \frac{\sigma^2}{4}) = \frac{{\epsilon_\text{old}}_i}{2}$. From there, we can conclude that $\mathbf{\epsilon}_\text{new} = \frac{\mathbf{\epsilon}_\text{old}}{2}$.

Does this somehow translate into the variance term $(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{\epsilon}$? I'm struggling to see how $\lambda_1/\lambda_2 = 4$, as this would imply that $\lambda_\text{new} = \frac{\lambda_1}{4}$. Is there some form of correlation between the distribution that $\epsilon$ is drawn from and $\lambda$?

Would appreciate any suggestions to approach this problem - please let me know if my logic is also off anywhere. Thanks in advance!

Answer 1

Maybe this problem is motivated by a connection to Bayesian Ridge Regression with assumptions $\mathbf y\,|\,\mathbf w, \sigma^2 \sim \mathcal N(\mathbf X \mathbf w, \sigma^2\mathbf I)$ and $\mathbf w \,|\, \tau^2 \sim \mathcal N(\mathbf 0, \tau^2\mathbf I)$. In this setting, maximizing the posterior density $p(\mathbf w\,|\, \mathbf y, \sigma^2, \tau^2)$ w.r.t. $\mathbf w$ for fixed $\sigma^2, \tau^2$ is equivalent to minimizing the Ridge-penalized least squares criterion for a fixed $\lambda \equiv \sigma^2/\tau^2$ since $$ \begin{align} p(\mathbf w\,|\, \mathbf y, \sigma^2, \tau^2)& \propto p(\mathbf y\,|\,\mathbf w, \sigma^2) \cdot p(\mathbf w \,|\, \tau^2)\\ & \propto \exp\left(-\frac{1}{2\sigma^2}\left\|\mathbf y - \mathbf X \mathbf w\right\|_2^2\right) \cdot \exp\left(-\frac{1}{2\tau^2}\left\|\mathbf w\right\|_2^2\right) \\ &=\exp\left(-\frac{1}{2\sigma^2}\left(\left\|\mathbf y - \mathbf X \mathbf w\right\|_2^2 + \frac{\sigma^2}{\tau^2}\left\|\mathbf w\right\|_2^2\right)\right). \end{align} $$ However, as pointed out in Sextus Empiricus' answer, this does not imply that when $\sigma^2$ changes the optimal (w.r.t. cross-validation) $\lambda$ has to change by the same factor.

Answer 2

The naive approach will be to use the cost function $$\sum_{i=1}^n (\hat{y} - y)^2 + \lambda \sum_{i=1}^p \beta_i^2$$ and consider that when the first term decreases by 4 because of the better sensor, then the second term can be decreased by 4 as well.

However, possibly it is not so trivial. When we decrease the penalty $\lambda$ by a factor $4$, then this will have a sort of feedback effect on the first term which initially decreased by $4$ but now will be larger again because the predictions $\hat{\beta}$ have changed.

What we can say is that the answer will be between two values $1< \lambda_1/\lambda_2 < 4$ but whether it is closer to $1$, $2$, or $4$ depends on the specific problem.