7
$\begingroup$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression coefficients $\hat\beta$ (aka the solution to $\arg\min_b \big[(y-Xb)'(y-Xb) + \lambda b'b\big]$) and their variance-covariance matrix $var(\hat\beta)$ are:

$$ \begin{align*} M &:= (X'X + \lambda I)^{-1}X' \\ \hat\beta &= My \\ var(\hat\beta) &= \sigma^2MM' \end{align*} $$

The computation on $var(\hat\beta)$ relies on $\lambda$ being treated as a constant value, which makes $M$ constant. Thus, we can apply the identity $var(My) = Mvar(y)M'$ and the fact that $var(y) = \sigma^2I$ to derive $var(\hat\beta)$.

However when $\lambda$ is selected using $X$ and $y$ (for my application via cross-validation), $\lambda$ and $M$ become stochastic. How can $var(\hat\beta)$ be updated to deal with a stochastic $\lambda$ selected via cross-validation?

$\endgroup$
1
$\begingroup$

I'm guessing that these equations are maximum likelihood solutions. The MLE of a parameter takes as its variance-covariance matrix the inverse of the second derivative of the likelihood function. What this means is that var$(\hat B)\sim∂^2L($data, $\hat B)/∂B^2$. If you include the lambda parameter, this partial derivative does not change. You introduce a covariance between B and lambda. The fact that you're optimizing lambda via cross-validation rather than MLE doesn't change the story for beta.

$\endgroup$
  • $\begingroup$ I've updated my question to clarify how $\hat\beta$ and $var(\hat\beta)$ are derived -- $\hat\beta$ is selected to minimize the sum of squared residuals plus a penalty term, and $var(\hat\beta)$ is based on a simple identity that relies on constant $\lambda$ values. I don't understand link to MLE that you're drawing in this answer -- could you clarify? $\endgroup$ – josliber Jul 9 '14 at 0:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.