In linear least squares the parameter estimates are: $\hat{\beta} = \left(X^{\top}X\right)^{-1}X^{\top}y$. In Ridge regression the standardized parameter estimates are given by $\hat{\beta}_{\Gamma} = \left(X^{\top}X + \Gamma\right)^{-1}X^{\top}y$.

Variance covariance estimators use leverage values to account for bias in the variance covariance estimates. For linear least squares a few examples include HC2 and HC3 which have effective residuals as: $\tilde{u} = \frac{\hat{u}}{1 - h_{ii}}$ and $\tilde{u} = \frac{\hat{u}}{\left(1 - h_{ii}\right)^{2}}$. $h_{ii}$ are the diagonal values of the projection matrix of the model matrix $h = X\left(X^{\top}X\right)^{-1}X^{\top}$.

My question is: for Ridge Regression, would the leverage adjusted residuals use $h = X\left(X^{\top}X\right)^{-1}X^{\top}$ or do they use a different for due to the fact that

$\hat{y}_{OLS} = X\left(X^{\top}X\right)^{-1}X^{\top}y \ne \left(X^{\top}X + \Gamma\right)^{-1}X^{\top}y = \hat{y}_{\Gamma}$? If different what is the form? How does it generalizes to clusters?


1 Answer 1


Ridge regression can be calculated via ordinary least squares (OLS) calculated with the data matrix $X$ extended with some surrogate data, taken as corresponding to the surrogate observations $Y_0=0$. Write the model, extended with the surrogate data, as $$ \begin{pmatrix} Y \\ Y_0=0\end{pmatrix} = \begin{pmatrix} X\beta \\ X_0 \beta \end{pmatrix} + \begin{pmatrix} \epsilon \\ \epsilon_0 \end{pmatrix} $$ Using this surrogate formulation, we can calculate the USUAL OLS estimator as $$ ( X^TX + X_0^T X_0)^{-1} (X^T Y + X_0^T 0) = (X^T X + X_0^T X_0)^{-1}X^TY $$ and comparing that with your expression for the ridge estimator, shows that you need to solve $$ \Gamma = X_0^T X_0 $$ for $X_0$, any matrix square root will do, for instance the Cholesky decomposition.

Then you can use the usual formulas for leverage with the extended data matrix $$ \begin{pmatrix} X \\ X_0 \end{pmatrix} $$ which is an $(n+p)\times p$-matrix, so the $n$ first leverage values correspond to the data. As a bonus you get, from the $p$ last leverage values, leverage information on the surrogate data, so you get information on how much influence there is from the surrogate data.

You ask also about cluster robust versions. I don't know about those, but guess the same approach can be used.

  • 1
    $\begingroup$ Obvious +1. The ridge regression by data augmentation is one my favourite procedures! It is worth noting that linear mixed effect regression models can be formulated in a similar manner. :D $\endgroup$
    – usεr11852
    Commented Nov 30, 2017 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.