Matrix Inversion Error

Question

I a Multiple linear regression model, from published literature, I am implementing a spreadsheet to generate new predictions based on the published model. the literature stated Coefficients and the error in each as well as the RMS error. However to estimate prediction interval i need to know the matrix (X'X)^-1 Where X is a k+1 by N model matrix. and X' it the transpose of X

As per http://www.stats.uwo.ca/faculty/braun/ss3859/chapters/chapter_4/ch4.pdf (Slide 45)

So, I downloaded the training set of data from the supplementary data, and tried to make my own X matrix. However, I have tried inverting (X'X) in both excel and R without success. Excel gives me garbage output while R gives me a singularity error. However, if I eliminate the "column of one's" from the X matrix making it an k by X matrix, now (X'X) inverts just fine.

Just to see what is going on, I've also tried feeding the training set of points into Excel's Linnest function as well as R's lm , and I get a model out of both with co-efficient s that are very close to the published literature models. So this makes me think the matrix shouldn't have any issues like this.

Also note, the model has 5 quantitative and 1 qualitative dimension (with 42 possible results), so final model has 49 dimensions.

So I am wondering where I went wrong? why is my X'X matrix not invertible when I include the column of ones?

Thanks

Answer 1

Assume $X$ is a $N \times M$ matrix. $X' X$ is singular if $X$'s rank is less than $\min(N, M)$: then resulting matrix will have rank less than $M$ and thus will be non-invertable.

Rank is sort of measure of how much unique data you have in the matrix. Why $X$ would have rank less than $M$ in the first place? One reason is feature correlations: if you have two identical columns, then you, essentially, have information about only one of them.

From geometrical perspective you can think of a matrix equation $A x = b$ where $A$ is a $N \times M$ matrix and $x \in \mathbb{R}^M$ and $b \in \mathbb{R}^N$ are vectors. Then each row of A defines a hyperplane in $\mathbb{R}^M$. Rank is number of independent hyperplanes. You need at least $M$ such (independent) hyperplanes to get a single point where they all intersect. If you have less independent hyperplanes, you'll get not a single point as a solution, but a whole hyperplane (line, if $rank(A) = M-1$ of them. If $A$ is square matrix, then solution is given using inverse of $A$: $x = A^{-1} b$. So if $A$ is of full-rank (that is, $rank(A)=M$), then solution is unique.

In you case $X' X$ (which is square matrix $M \times M$) will be of full-rank if and only if $rank(X) = M$

So, what can you do? There are 2 possibilities:

1. Find and eliminate any duplicating columns in your data.
2. Add a Tikhonov regularization (see Ridge Regression)