# Prove that the variance of the ridge regression estimator is less than the variance of the OLS estimator

Consider the following linear model under classical Gauss-Markov assumtions: $$Y = X\beta + e$$ where $\mathbb{E}X'e = 0$

Consider the following estimator $$\tilde\beta = \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i+1}^Nx_iy_i\right)$$ where $x_i$ is a column vector $k\times1$ from $X$ and $\lambda > 0$ is a scalar and $\mathbb{E}(x_ie_i) = 0$ .

1. Define bias and show that $\tilde\beta$ is biased.
2. Define consistency and show that $\tilde\beta$ is consistent.
3. Define conditional variance of $\tilde\beta$. Show that conditional variance of $\tilde\beta$ is smaller then the conditional variance of OLS estimator $\hat\beta$.
4. Give two reasons why we want to prefer using $\tilde\beta$ instead of $\hat\beta$. (Hint: think of collinearity).

First two questions are answered (with the help of Cross Validated).

Define $\left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1} = (X'X + \lambda I)^{-1} = W$. Also note that under homoskedasticity $Var(\hat\beta) = \sigma^2(X'X)^{-1}$. For the third one I have \begin{aligned} Var(\tilde\beta|X) &= Var(WX'Y|X) \\ & = WX'Var(Y|X)XW \\ & = WX'Var(X\beta + u|X)XW \\ & = WX'Var(u|X)XW \\ \text{(assuming homoskedasticity)}& = WX'\sigma^2IXW \\ & = \sigma^2WX'XW \end{aligned}

Now to end with question 3 I need to show that $(X'X)^{-1} - WX'XW$ is positive semidefinite. This is the place where I am stuck. I also have no ideas on question 4.

EDIT: please note that this is question from the last years exam which almost surely means that the question can be solved using basic matrix algebra and not more advanced technics like SVD etc.

• Calculate the derivative wrt $\lambda$ and evaluate at zero ... Nov 19, 2016 at 20:09
• @kjetil b halvorsen, thanks for your comment. I frankly don't know how to take derivative of the expression like $(X'X - \lambda I)^{-1}X'X(X'X + \lambda I)^{-1}$ wrt to $\lambda$ (if I understand you right) and also don't see how it can help. Nov 19, 2016 at 20:23
• Hint: Set $S = X'X$ and note that $$S^{-1} - W S W = WW^{-1}S^{-1}W^{-1}W - WSW = W (W^{-1}S^{-1}W^{-1} - S)W,$$ and simplify the middle matrix using $W^{-1} = S + \lambda I$. Conclusion? Nov 20, 2016 at 2:07
• @cardinal, could you check please (see below). Nov 20, 2016 at 20:19

According to the cardinal's hint. We want to show that $(X'X)^{-1} - WX'XW$ is psd. Denote $X'X = S$. Then, $S^{-1} - WSW = WW^{-1}S^{-1}W^{-1}W - WSW= W(W^{-1}S^{-1}W^{-1} - S)W$. Take expression in the brackets and simplify \begin{aligned} W^{-1}S^{-1}W^{-1} - S &= (S+\lambda I)S^{-1}(S+\lambda ) - S \\ & = SS^{-1}S + SS^{-1}\lambda +\lambda S^{-1}S + \lambda^2S^{-1} - S\\ & = 2\lambda I + \lambda^2S^{-1} \\ & = \lambda(2I + \lambda S^{-1}). \end{aligned} Since $S^{-1}$ is psd then the whole expression is psd matrix.

Then original expression can be represented as $$W(W^{-1}S^{-1}W^{-1} - S)W = \lambda W(2I + \lambda S^{-1})W$$ which must be psd since expression in the brackets is positive semi-definite matrix.

We can write $W = \lambda^{-1} (\lambda^{-1}X'X + I)^{-1}$. Set for compactness $P\equiv\lambda^{-1} X'X$. Then you want to examine the expression

$$\lambda^{-1}P^{-1} - \lambda^{-1}(P+I)^{-1}\lambda P\lambda^{-1}(P+I)^{-1}$$

and you can simplify and ignore $\lambda^{-1}$ (which is positive). So we are examining

$$P^{-1} - (P+I)^{-1}P(P+I)^{-1} = (P+I)^{-1}\Big[(P+I)P^{-1} - P(P+I)^{-1}\Big]$$

$$=(P+I)^{-1}\Big[ I + P^{-1}- P(P+I)^{-1}\Big]$$

From what I know as the "Searl set of identities" related to inverse matrices we have $I- P(P+I)^{-1}= (P+I)^{-1}$ so we get

$$(P+I)^{-1}\Big[P^{-1} +(P+I)^{-1}\Big] = (P+I)^{-1}P^{-1} + (P+I)^{-1}(P+I)^{-1}$$

$$= (PP+P)^{-1} + (P+I)^{-1}(P+I)^{-1}$$

The sum of two positive definite matrices is positive definite. The inverse of a positive definite matrix is pd. The product of two positive definite matrices is also positive definite if the matrices commute i.e. $AB = BA$.

$PP$ commutes so $PP$ is positive definite and so is then $(PP+P)^{-1}$. Also, $(P+I)^{-1}(P+I)^{-1}$ commutes, and so this product also is pd. So both components of this sum are pd so the sum is also pd. QED.

• Thanks for your answer (+1). Do you have any thoughts about question 4. Namely, why we want to prefer using $\tilde\beta$ instead of usual OLS estimator? Nov 20, 2016 at 20:21
• @TOSIK When both are available (in the sense that OLS does not run into collinearity etc problems), then one could look at mean-squared error : many times experts here have expressed a willingness to tolerate the existence of bias in return for smaller variance. Formally, one should attempt to see whether the ridge estimator has smaller mean-squared error (i.e. variance plus squared bias). Nov 20, 2016 at 20:25
• I am asked to think about collinearity. How does this estimator helps to tackle this problem. As far as I concerned, collinearity may result into big standard errors. Is it correct to say that possible reason to use ridge estimator is to solve this consequence of collinearity? Nov 20, 2016 at 20:30
• @tosik Look up the literature or even web references, this is one of the main reasons to apply ridge regression. Nov 20, 2016 at 20:31

As $$A\geq B\Leftrightarrow B^{-1}\geq A^{-1}$$, we may also establish \begin{align*} W^{-1}(X'X)^{-1}W^{-1}-X'X\geq0 \end{align*} Plug in to get \begin{align*} (X'X+\lambda I)(X'X)^{-1}(X'X+\lambda I)-X'X\geq0. \end{align*} Multiplying out and collecting terms gives \begin{align*} 2\lambda I+\lambda^2(X'X)^{-1}\geq0, \end{align*} which is true because $$\lambda>0$$, $$X'X$$ is p.s.d and hence so is its inverse as is the identity matrix. Also, the sum of two p.s.d. matrices is also p.s.d.

Here is a numerical example, in which the regressor has relatively little variation. Hence, the ridge correction matters relatively much (e.g., for a simple regression without intercept model we have $$\hat\beta_{\text{ridge}}/\hat\beta_{\text{OLS}}=(1+\lambda/\sum_ix_i^2)^{-1}$$). I plot the slopes of least squares (coral colors) and ridge (green) of 50 replications of the experiment against a representative draw from the DGP.

We observe that, while least squares is, unlike ridge, "correct on average" (unbiased) for the true slope (blue, equal to two), it is rather volatile. Ridge, in turn, is more stable - albeit, being biased, around a wrong value.

ls.vs.ridge <- function(n){
x <- matrix(cbind(1, rnorm(n, sd=.06)), ncol=2)
u <- rnorm(n)
y <- 2*x[,2] + u

limo <- lm(y~x[,2])

lambda <- 2
ridge <- solve(crossprod(x)+lambda*diag(2))%*%crossprod(x,y)
return(c(coef(limo), ridge))
}

n <- 200
reps <- 50
ls.ridge <- replicate(reps, ls.vs.ridge(n))

x <- matrix(cbind(1, rnorm(n, sd=.06)), ncol=2)
u <- rnorm(n)
y <- 2*x[,2] + u
plot(x[,2], y, xlim=c(-1,1))

for (i in 1:reps){
abline(a = ls.ridge[1,i], b = ls.ridge[2,i], col="coral", lwd=1)
abline(a = ls.ridge[3,i], b = ls.ridge[4,i], col="darkolivegreen3", lwd=1)
}
abline(a = 0, b = 2, col="blue", lwd=3)
mean(ls.ridge[2,])
mean(ls.ridge[4,])