4
$\begingroup$

Currently reviewing some problems in my ML class, and I came across this problem:

You estimate a ridge regression model with some data taken from your robot, and find (using cross validation) and optimal ridge penalty $\lambda_1$. You then buy a new sensor which has noise with $1/4$ the variance (half the standard deviation) as before. Using the same number of observations as before you collect new data, and find a new optimal ridge penalty $\lambda_2$.

Which of the following will be closest to true?

  1. $\lambda_1 / \lambda_2 = 1/4$
  2. $\lambda_1 / \lambda_2 = 1/2$
  3. $\lambda_1 / \lambda_2 = 1$
  4. $\lambda_1 / \lambda_2 = 2$
  5. $\lambda_1 / \lambda_2 = 4$

The solution key indicates that the answer is $\lambda_1 / \lambda_2 = 4$, but does not provide an explanation.

Here is my attempt at solving the problem:

From an assumption that $y_i = \mathbf{x}_i^\top \mathbf{w}^* + \epsilon_i$ where each error epsilon is drawn from a gaussian distribution with 0 mean and variance $\sigma^2$ (i.e. $\epsilon_i \sim N(0, \sigma^2)$), we have that ridge regression is

$$ \widehat{\mathbf{w}}_\text{ridge} = \arg\underset{\mathbf{w}}{\min} \sum_{i=1}^{n}(y_i - \mathbf{x}_i^\top \mathbf{w})^2 + \lambda \|\mathbf{w} \|_2^2 $$

The closed form solution for $\widehat{\mathbf{w}}_\text{ridge}$ is $$ \widehat{\mathbf{w}}_\text{ridge} = (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y} $$

When calculating the bias-variance tradeoff, we get $$ \begin{aligned} \widehat{\mathbf{w}}_\text{ridge} &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top(\mathbf{X}\mathbf{w}^* + \mathbf{\epsilon}) && \mathbf{y} = \mathbf{X}\mathbf{w}^* + \mathbf{\epsilon} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}(\mathbf{X}^\top\mathbf{X}\mathbf{w}^* + \mathbf{X}^\top\mathbf{\epsilon}) \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}(\mathbf{X}^\top\mathbf{X}\mathbf{w}^* + \mathbf{X}^\top\mathbf{\epsilon} + \lambda\mathbf{I{w^*}} - \lambda\mathbf{I{w^*}}) &&\text{adding zero} \\ &= (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}((\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})\mathbf{w}^* - \lambda\mathbf{I{w^*}} + \mathbf{X}^\top\mathbf{\epsilon}) &&\text{rearranging terms} \\ &= \mathbf{w}^* - (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\lambda\mathbf{w}^* + (\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{\epsilon} &&\text{distribute} \end{aligned} $$

If our sensor has noise with $1/4$ the variance, then our new error ${\epsilon_\text{new}}_i \sim N(0, \frac{\sigma^2}{4}) = \frac{{\epsilon_\text{old}}_i}{2}$. From there, we can conclude that $\mathbf{\epsilon}_\text{new} = \frac{\mathbf{\epsilon}_\text{old}}{2}$.

Does this somehow translate into the variance term $(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{\epsilon}$? I'm struggling to see how $\lambda_1/\lambda_2 = 4$, as this would imply that $\lambda_\text{new} = \frac{\lambda_1}{4}$. Is there some form of correlation between the distribution that $\epsilon$ is drawn from and $\lambda$?

Would appreciate any suggestions to approach this problem - please let me know if my logic is also off anywhere. Thanks in advance!

$\endgroup$
1
  • $\begingroup$ The question implicitly assumes all the variation in the response is sensor "noise." $\endgroup$
    – whuber
    Commented Nov 8, 2022 at 13:30

2 Answers 2

4
$\begingroup$

Maybe this problem is motivated by a connection to Bayesian Ridge Regression with assumptions $\mathbf y\,|\,\mathbf w, \sigma^2 \sim \mathcal N(\mathbf X \mathbf w, \sigma^2\mathbf I)$ and $\mathbf w \,|\, \tau^2 \sim \mathcal N(\mathbf 0, \tau^2\mathbf I)$. In this setting, maximizing the posterior density $p(\mathbf w\,|\, \mathbf y, \sigma^2, \tau^2)$ w.r.t. $\mathbf w$ for fixed $\sigma^2, \tau^2$ is equivalent to minimizing the Ridge-penalized least squares criterion for a fixed $\lambda \equiv \sigma^2/\tau^2$ since $$ \begin{align} p(\mathbf w\,|\, \mathbf y, \sigma^2, \tau^2)& \propto p(\mathbf y\,|\,\mathbf w, \sigma^2) \cdot p(\mathbf w \,|\, \tau^2)\\ & \propto \exp\left(-\frac{1}{2\sigma^2}\left\|\mathbf y - \mathbf X \mathbf w\right\|_2^2\right) \cdot \exp\left(-\frac{1}{2\tau^2}\left\|\mathbf w\right\|_2^2\right) \\ &=\exp\left(-\frac{1}{2\sigma^2}\left(\left\|\mathbf y - \mathbf X \mathbf w\right\|_2^2 + \frac{\sigma^2}{\tau^2}\left\|\mathbf w\right\|_2^2\right)\right). \end{align} $$ However, as pointed out in Sextus Empiricus' answer, this does not imply that when $\sigma^2$ changes the optimal (w.r.t. cross-validation) $\lambda$ has to change by the same factor.

$\endgroup$
1
$\begingroup$

The naive approach will be to use the cost function $$\sum_{i=1}^n (\hat{y} - y)^2 + \lambda \sum_{i=1}^p \beta_i^2$$ and consider that when the first term decreases by 4 because of the better sensor, then the second term can be decreased by 4 as well.

However, possibly it is not so trivial. When we decrease the penalty $\lambda$ by a factor $4$, then this will have a sort of feedback effect on the first term which initially decreased by $4$ but now will be larger again because the predictions $\hat{\beta}$ have changed.

What we can say is that the answer will be between two values $1< \lambda_1/\lambda_2 < 4$ but whether it is closer to $1$, $2$, or $4$ depends on the specific problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.