Prove that the variance of the ridge regression estimator is less than the variance of the OLS estimator Consider the following linear model under classical Gauss-Markov assumtions:
$$Y = X\beta + e$$
where $\mathbb{E}X'e = 0$
Consider the following estimator
$$\tilde\beta = \left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1}\left(\sum_{i+1}^Nx_iy_i\right)$$
where $x_i$ is a column vector $k\times1$ from $X$ and $\lambda > 0$ is a scalar and $\mathbb{E}(x_ie_i) = 0$ .


*

*Define bias and show that $\tilde\beta$ is biased.

*Define consistency and show that $\tilde\beta$ is consistent.

*Define conditional variance of $\tilde\beta$. Show that conditional variance of $\tilde\beta$ is smaller then the conditional variance of OLS estimator $\hat\beta$.

*Give two reasons why we want to prefer using $\tilde\beta$ instead of $\hat\beta$. (Hint: think of collinearity).


First two questions are answered (with the help of Cross Validated).
Define $\left(\sum_{i=1}^{N}x_ix_i' + \lambda I_k\right)^{-1} = (X'X + \lambda I)^{-1} = W$. Also note that under homoskedasticity $Var(\hat\beta) = \sigma^2(X'X)^{-1}$.
For the third one I have
\begin{equation}
\begin{aligned}
Var(\tilde\beta|X) &= Var(WX'Y|X) \\
& = WX'Var(Y|X)XW \\
& = WX'Var(X\beta + u|X)XW \\
& = WX'Var(u|X)XW \\
\text{(assuming homoskedasticity)}& = WX'\sigma^2IXW \\
& = \sigma^2WX'XW
\end{aligned}
\end{equation}
Now to end with question 3 I need to show that $(X'X)^{-1} - WX'XW$ is positive semidefinite. This is the place where I am stuck. I also have no ideas on question 4. 

EDIT: please note that this is question from the last years exam which almost surely means that the question can be solved using basic matrix algebra and not more advanced technics like SVD etc. 
 A: According to the cardinal's hint. We want to show that $(X'X)^{-1} - WX'XW$ is psd. Denote $X'X = S$. Then, $S^{-1} - WSW = WW^{-1}S^{-1}W^{-1}W - WSW= W(W^{-1}S^{-1}W^{-1} - S)W$. Take expression in the brackets and simplify
\begin{equation}
\begin{aligned}
W^{-1}S^{-1}W^{-1} - S &= (S+\lambda I)S^{-1}(S+\lambda ) - S \\
& = SS^{-1}S + SS^{-1}\lambda +\lambda S^{-1}S + \lambda^2S^{-1} - S\\
& = 2\lambda I + \lambda^2S^{-1} \\
& = \lambda(2I + \lambda S^{-1}).
\end{aligned} 
\end{equation}
Since $S^{-1}$ is psd then the whole expression is psd matrix. 
Then original expression can be represented as 
$$W(W^{-1}S^{-1}W^{-1} - S)W = \lambda W(2I + \lambda S^{-1})W$$ 
which must be psd since expression in the brackets is positive semi-definite matrix. 
A: We can write $W = \lambda^{-1} (\lambda^{-1}X'X + I)^{-1}$. Set for compactness $P\equiv\lambda^{-1} X'X$. Then you want to examine the expression
$$\lambda^{-1}P^{-1} - \lambda^{-1}(P+I)^{-1}\lambda P\lambda^{-1}(P+I)^{-1}$$
and you can simplify and ignore $\lambda^{-1}$ (which is positive). So we are examining
$$P^{-1} - (P+I)^{-1}P(P+I)^{-1} = (P+I)^{-1}\Big[(P+I)P^{-1} - P(P+I)^{-1}\Big]$$
$$=(P+I)^{-1}\Big[ I + P^{-1}- P(P+I)^{-1}\Big]$$
From what I know as the "Searl set of identities" related to inverse matrices we have  $I- P(P+I)^{-1}= (P+I)^{-1}$ so we get
$$(P+I)^{-1}\Big[P^{-1} +(P+I)^{-1}\Big] = (P+I)^{-1}P^{-1} + (P+I)^{-1}(P+I)^{-1} $$
$$= (PP+P)^{-1} + (P+I)^{-1}(P+I)^{-1}$$
The sum of two positive definite matrices is positive definite. The inverse of a positive definite matrix is pd. The product of two positive definite matrices is also positive definite if the matrices commute i.e. $AB = BA$. 
$PP$ commutes so $PP$ is positive definite and so is then $(PP+P)^{-1}$. Also, $(P+I)^{-1}(P+I)^{-1}$ commutes, and so this product also is pd. So both components of this sum are pd so the sum is also pd. QED. 
A: As $A\geq B\Leftrightarrow B^{-1}\geq A^{-1}$, we may also establish
\begin{align*}
W^{-1}(X'X)^{-1}W^{-1}-X'X\geq0
\end{align*}
Plug in to get
\begin{align*}
(X'X+\lambda I)(X'X)^{-1}(X'X+\lambda I)-X'X\geq0.
\end{align*}
Multiplying out and collecting terms gives
\begin{align*}
2\lambda I+\lambda^2(X'X)^{-1}\geq0,
\end{align*}
which is true because $\lambda>0$, $X'X$ is p.s.d and hence so is its inverse as is the identity matrix. Also, the sum of two p.s.d. matrices is also p.s.d.
Here is a numerical example, in which the regressor has relatively little variation. Hence, the ridge correction matters relatively much (e.g., for a simple regression without intercept model we have $\hat\beta_{\text{ridge}}/\hat\beta_{\text{OLS}}=(1+\lambda/\sum_ix_i^2)^{-1}$). I plot the slopes of least squares (coral colors) and ridge (green) of 50 replications of the experiment against a representative draw from the DGP.
We observe that, while least squares is, unlike ridge, "correct on average" (unbiased) for the true slope (blue, equal to two), it is rather volatile. Ridge, in turn, is more stable - albeit, being biased, around a wrong value.

ls.vs.ridge <- function(n){
  x <- matrix(cbind(1, rnorm(n, sd=.06)), ncol=2)
  u <- rnorm(n)
  y <- 2*x[,2] + u
  
  limo <- lm(y~x[,2])
  
  lambda <- 2
  ridge <- solve(crossprod(x)+lambda*diag(2))%*%crossprod(x,y)
  return(c(coef(limo), ridge))
}

n <- 200
reps <- 50
ls.ridge <- replicate(reps, ls.vs.ridge(n))

x <- matrix(cbind(1, rnorm(n, sd=.06)), ncol=2)
u <- rnorm(n)
y <- 2*x[,2] + u
plot(x[,2], y, xlim=c(-1,1))

for (i in 1:reps){
  abline(a = ls.ridge[1,i], b = ls.ridge[2,i], col="coral", lwd=1)
  abline(a = ls.ridge[3,i], b = ls.ridge[4,i], col="darkolivegreen3", lwd=1)
}
abline(a = 0, b = 2, col="blue", lwd=3)
mean(ls.ridge[2,])
mean(ls.ridge[4,])

