Bias variance tradeoff of ridge regression with independent but non identically distributed error?

Question

I am trying to figure out how the solution for ridge regression changes when the error term is independent but NOT identically distributed such as $\mathbb\epsilon = \mathcal{N}(0, \Sigma)$ rather than $\mathbb\epsilon = \mathcal{N}(0, \sigma^{2})$

In standard ridge regression we have:

$$\hat \beta(\lambda) = \arg\min_{\beta}\left\{\frac 1 n \sum_{i=1}^n (y_i -x_i^T \beta)^2 + \lambda \|\beta\|_2^2\right\}$$

which can be solved as:

$$\hat\beta(\lambda) = (X^TX + \lambda I)^{-1} X^T y$$

when we use $\mathbb\epsilon = \mathcal{N}(0, \sigma^{2})$.

I am struggling with how to determine the difference when one would assume $\mathbb\epsilon = \mathcal{N}(0, \Sigma)$ instead. My initial thought was it would be:

$$\hat\beta(\lambda) = (X^TX + \lambda \Sigma)^{-1} X^T y$$

but I think I am off track there. In the end I want to calculate MSE, Bias and Varience using this alternate error distribution as described here,

Ridge Regression: how to show squared bias increases as $\lambda$ increases

Answer 1

As JohnK noted, the optimization problem that defines the ridge regression estimator can be defined without making any distributional assumptions on the noise; in particular the solution $\hat{\beta}_r = (X^\top X + \lambda I)^{-1} X^\top y$ remains the same.

I'll use the same notation $K = (X^\top X + \lambda)^{-1} X^\top X$ and $\hat{\beta} = (X^\top X)^{-1} X^\top y$ as in the answer you linked.

The bias term does not change, since the only information about $\epsilon$ that you use in that computation is $E[\epsilon] = 0$.

The variance term does depend on $\Sigma$. Specifically, $$\hat{\beta}_r - E[\hat{\beta}_r] = K(\hat{\beta} - E[\hat{\beta}]) = K(X^\top X)^{-1} X^\top \epsilon$$ and \begin{align} E[(\hat{\beta}_r - E[\hat{\beta}_r])^\top (\hat{\beta}_r - E[\hat{\beta}_r])] &= E[\text{tr}((\hat{\beta}_r - E[\hat{\beta}_r])(\hat{\beta}_r - E[\hat{\beta}_r])^\top)] \\ &= E[\text{tr}(K(X^\top X)^{-1} X^\top \epsilon \epsilon^\top X (X^\top X)^{-1} K^\top)] \\ &= \text{tr}(K(X^\top X)^{-1} X^\top E[\epsilon \epsilon^\top] X(X^\top X)^{-1} K^\top) \\ &= \text{tr}(K(X^\top X)^{-1} X^\top \Sigma X(X^\top X)^{-1} K^\top). \end{align} When $\Sigma = \sigma^2 I$, you can simplify the last expression to $\sigma^2 \text{tr}(K(X^\top X)^{-1} K^\top)$ which is the variance mentioned in the answer you linked. But for more general $\Sigma$ I don't think there is a nice simplification beyond the above.

Answer 2

To be clear, the solution to the ridge problem

$$ \min_{\beta}\left\{\frac 1 n \sum_{i=1}^n (y_i -x_i^T \beta)^2 + \lambda \|\beta\|_2^2\right\}$$

does not depend on the distribution of the error. Use, e.g., standard calculus to see this. This means that you do not have to modify the estimator if the error is not identically distributed. Sure, its theoretical properties may change but the estimator will remain the same.