# Prove that the variance of the ridge regression estimator is less than the variance of the OLS estimator

Consider the following linear model under classical Gauss-Markov assumtions: $$Y = X\beta + e$$ where $\mathbb{E}X'e = 0$

Consider the following estimator $$\tilde\beta = \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i+1}^Nx_iy_i\right)$$ where $x_i$ is a column vector $k\times1$ from $X$ and $\lambda > 0$ is a scalar and $\mathbb{E}(x_ie_i) = 0$ .

1. Define bias and show that $\tilde\beta$ is biased.
2. Define consistency and show that $\tilde\beta$ is consistent.
3. Define conditional variance of $\tilde\beta$. Show that conditional variance of $\tilde\beta$ is smaller then the conditional variance of OLS estimator $\hat\beta$.
4. Give two reasons why we want to prefer using $\tilde\beta$ instead of $\hat\beta$. (Hint: think of collinearity).

First two questions are answered (with the help of Cross Validated).

Define $\left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1} = (X'X + \lambda I)^{-1} = W$. Also note that under homoskedasticity $Var(\hat\beta) = \sigma^2(X'X)^{-1}$. For the third one I have \begin{aligned} Var(\tilde\beta|X) &= Var(WX'Y|X) \\ & = WX'Var(Y|X)XW \\ & = WX'Var(X\beta + u|X)XW \\ & = WX'Var(u|X)XW \\ \text{(assuming homoskedasticity)}& = WX'\sigma^2IXW \\ & = \sigma^2WX'XW \end{aligned}

Now to end with question 3 I need to show that $(X'X)^{-1} - WX'XW$ is positive semidefinite. This is the place where I am stuck. I also have no ideas on question 4.

EDIT: please note that this is question from the last years exam which almost surely means that the question can be solved using basic matrix algebra and not more advanced technics like SVD etc.

• Calculate the derivative wrt $\lambda$ and evaluate at zero ... – kjetil b halvorsen Nov 19 '16 at 20:09
• @kjetil b halvorsen, thanks for your comment. I frankly don't know how to take derivative of the expression like $(X'X - \lambda I)^{-1}X'X(X'X + \lambda I)^{-1}$ wrt to $\lambda$ (if I understand you right) and also don't see how it can help. – tosik Nov 19 '16 at 20:23
• Hint: Set $S = X'X$ and note that $$S^{-1} - W S W = WW^{-1}S^{-1}W^{-1}W - WSW = W (W^{-1}S^{-1}W^{-1} - S)W,$$ and simplify the middle matrix using $W^{-1} = S + \lambda I$. Conclusion? – cardinal Nov 20 '16 at 2:07
• @cardinal, could you check please (see below). – tosik Nov 20 '16 at 20:19

According to the cardinal's hint. We want to show that $(X'X)^{-1} - WX'XW$ is psd. Denote $X'X = S$. Then, $S^{-1} - WSW = WW^{-1}S^{-1}W^{-1}W - WSW= W(W^{-1}S^{-1}W^{-1} - S)W$. Take expression in the brackets and simplify \begin{aligned} W^{-1}S^{-1}W^{-1} - S &= (S+\lambda I)S^{-1}(S+\lambda ) - S \\ & = SS^{-1}S + SS^{-1}\lambda +\lambda S^{-1}S + \lambda^2S^{-1} - S\\ & = 2\lambda I + \lambda^2S^{-1} \\ & = \lambda(2I + \lambda S^{-1}). \end{aligned} Since $S^{-1}$ is psd then the whole expression is psd matrix.

Then original expression can be represented as $$W(W^{-1}S^{-1}W^{-1} - S)W = \lambda W(2I + \lambda S^{-1})W$$ which must be psd since expression in the brackets is positive semi-definite matrix.

We can write $W = \lambda^{-1} (\lambda^{-1}X'X + I)^{-1}$. Set for compactness $P\equiv\lambda^{-1} X'X$. Then you want to examine the expression

$$\lambda^{-1}P^{-1} - \lambda^{-1}(P+I)^{-1}\lambda P\lambda^{-1}(P+I)^{-1}$$

and you can simplify and ignore $\lambda^{-1}$ (which is positive). So we are examining

$$P^{-1} - (P+I)^{-1}P(P+I)^{-1} = (P+I)^{-1}\Big[(P+I)P^{-1} - P(P+I)^{-1}\Big]$$

$$=(P+I)^{-1}\Big[ I + P^{-1}- P(P+I)^{-1}\Big]$$

From what I know as the "Searl set of identities" related to inverse matrices we have $I- P(P+I)^{-1}= (P+I)^{-1}$ so we get

$$(P+I)^{-1}\Big[P^{-1} +(P+I)^{-1}\Big] = (P+I)^{-1}P^{-1} + (P+I)^{-1}(P+I)^{-1}$$

$$= (PP+P)^{-1} + (P+I)^{-1}(P+I)^{-1}$$

The sum of two positive definite matrices is positive definite. The inverse of a positive definite matrix is pd. The product of two positive definite matrices is also positive definite if the matrices commute i.e. $AB = BA$.

$PP$ commutes so $PP$ is positive definite and so is then $(PP+P)^{-1}$. Also, $(P+I)^{-1}(P+I)^{-1}$ commutes, and so this product also is pd. So both components of this sum are pd so the sum is also pd. QED.

• Thanks for your answer (+1). Do you have any thoughts about question 4. Namely, why we want to prefer using $\tilde\beta$ instead of usual OLS estimator? – tosik Nov 20 '16 at 20:21
• @TOSIK When both are available (in the sense that OLS does not run into collinearity etc problems), then one could look at mean-squared error : many times experts here have expressed a willingness to tolerate the existence of bias in return for smaller variance. Formally, one should attempt to see whether the ridge estimator has smaller mean-squared error (i.e. variance plus squared bias). – Alecos Papadopoulos Nov 20 '16 at 20:25
• I am asked to think about collinearity. How does this estimator helps to tackle this problem. As far as I concerned, collinearity may result into big standard errors. Is it correct to say that possible reason to use ridge estimator is to solve this consequence of collinearity? – tosik Nov 20 '16 at 20:30
• @tosik Look up the literature or even web references, this is one of the main reasons to apply ridge regression. – Alecos Papadopoulos Nov 20 '16 at 20:31

The key is to use : W >= X^tX imply that W^-1 <= X^tX and you can multiply by X^tX (S.D.P) no problem the inequality still work .... Note that (A >= B imply that A - B is S.D.P)

W >= X^tX imply that W^-1 <= (X^tX)^-1