1
$\begingroup$

There were many similar questions on this site , related to this but none were exactly to the point I wanted to ask

So the question is relates to ridge regression and This link where there is a statement(in bold) that

In a least squares fit where coefficient estimate given by

has no unique solution when is not invertible but

Inclusion of λ makes problem non-singular even if is not invertible

This was the original motivation for ridge regression (Hoerl and Kennard, 1970)

ie.

I am unable to exactly convince myself why this would be the case. Specifically for the case when p(predictors)>n(instances) how will I be able to obtain a unique solution with this?

$\endgroup$

marked as duplicate by Sycorax, Community May 31 '17 at 5:16

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

3
$\begingroup$

Full rank matrix is invertible. Adding $\lambda I$ to $X^TX$ make it a full rank matrix.

Here is an example. Suppose $X$ has two identical columns:

> x=cbind(c(1,1,1,1),c(1,1,1,1),c(1,2,3,4))
> x
     [,1] [,2] [,3]
[1,]    1    1    1
[2,]    1    1    2
[3,]    1    1    3
[4,]    1    1    4
> t(x) %*% x
     [,1] [,2] [,3]
[1,]    4    4   10
[2,]    4    4   10
[3,]   10   10   30
> Matrix::rankMatrix(t(x) %*% x)
[1] 2   
> Matrix::rankMatrix(t(x) %*% x + 1e6*diag(3))
[1] 3

$X^TX$ is a symmetric matrix. It is diagonalizable with real eigenvalues. Let those eigenvalues equal $\lambda_1, \cdots \lambda_n$.

If we choose any of the $\lambda_i$ above. $(X^TX - \lambda_i I)$ will create a singular matrix.

However, $(X^TX - \lambda I)$ where $\lambda \ne \lambda_1, \cdots \lambda_n$ then the eigenvalues of $(X^TX - \lambda I)$ equal $\lambda_1 - \lambda,\cdots,\lambda_n-\lambda$ none of which are equal to $0.$ Hence, $(X^TX - \lambda I)$ is non-singular.

Nearly any perturbation to a singular matrix will make for a non-singular matrix.

$\endgroup$
  • 2
    $\begingroup$ (+1)Yes, but would such adding always result in a full rank matrix? $\endgroup$ – GeneX May 31 '17 at 2:27
  • 1
    $\begingroup$ seem my edits on why. $\endgroup$ – Haitao Du Jun 1 '17 at 16:30
  • $\begingroup$ It should be noted that the second part of the answer was copied over from here. $\endgroup$ – Casimir Jan 2 at 12:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.