Non-Singularity due to inclusion of non-zero lambda in ridge regression [duplicate]

There were many similar questions on this site , related to this but none were exactly to the point I wanted to ask

So the question is relates to ridge regression and This link where there is a statement(in bold) that

In a least squares fit where coefficient estimate given by

has no unique solution when is not invertible but

Inclusion of λ makes problem non-singular even if is not invertible

This was the original motivation for ridge regression (Hoerl and Kennard, 1970)

ie.

I am unable to exactly convince myself why this would be the case. Specifically for the case when p(predictors)>n(instances) how will I be able to obtain a unique solution with this?

marked as duplicate by Sycorax, Community♦May 31 '17 at 5:16

Full rank matrix is invertible. Adding $\lambda I$ to $X^TX$ make it a full rank matrix.

Here is an example. Suppose $X$ has two identical columns:

> x=cbind(c(1,1,1,1),c(1,1,1,1),c(1,2,3,4))
> x
[,1] [,2] [,3]
[1,]    1    1    1
[2,]    1    1    2
[3,]    1    1    3
[4,]    1    1    4
> t(x) %*% x
[,1] [,2] [,3]
[1,]    4    4   10
[2,]    4    4   10
[3,]   10   10   30
> Matrix::rankMatrix(t(x) %*% x)
 2
> Matrix::rankMatrix(t(x) %*% x + 1e6*diag(3))
 3

$X^TX$ is a symmetric matrix. It is diagonalizable with real eigenvalues. Let those eigenvalues equal $\lambda_1, \cdots \lambda_n$.

If we choose any of the $\lambda_i$ above. $(X^TX - \lambda_i I)$ will create a singular matrix.

However, $(X^TX - \lambda I)$ where $\lambda \ne \lambda_1, \cdots \lambda_n$ then the eigenvalues of $(X^TX - \lambda I)$ equal $\lambda_1 - \lambda,\cdots,\lambda_n-\lambda$ none of which are equal to $0.$ Hence, $(X^TX - \lambda I)$ is non-singular.

Nearly any perturbation to a singular matrix will make for a non-singular matrix.

• (+1)Yes, but would such adding always result in a full rank matrix? – GeneX May 31 '17 at 2:27
• seem my edits on why. – Haitao Du Jun 1 '17 at 16:30
• It should be noted that the second part of the answer was copied over from here. – Casimir Jan 2 at 12:55