3
$\begingroup$

The ridge regression estimate is given by $$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$ where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for Nonorthogonal Problems, states that the eigenvalues, $\lambda_i$, of $X'X$ are related to eigenvalues, $\xi_i$, of $W = (X'X+kI)^{-1}$ as $\xi_i=1/(k+\lambda_i)$. This expression follows from solving the characteristic equation $|W - \xi_iI|=0$. I can only imagine using cofactor representation of the determinant. However, the inverse in $W$ complicates matters.

How exactly does one solve this characteristic equation?

$\endgroup$
6
$\begingroup$

If $\lambda$ is a nonzero eigenvalue of an invertible matrix $A$, then $\lambda^{-1}$ is an eigenvalue of $A^{-1}$. To show this, note that if $Av=\lambda v$ then $v = A^{-1}(\lambda v) = \lambda A^{-1} v$ and thus $\lambda^{-1} v = A^{-1} v$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Just want to point out that we don't need to assume that $\lambda$ is nonzero if we're already assuming that $A$ is invertible, as all eigenvalues of invertible matrices are nonzero. $\endgroup$ – Eric Perkerson Oct 4 at 19:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.