I a Multiple linear regression model, from published literature, I am implementing a spreadsheet to generate new predictions based on the published model. the literature stated Coefficients and the error in each as well as the RMS error. However to estimate prediction interval i need to know the matrix (X'X)^-1 Where X is a k+1 by N model matrix. and X' it the transpose of X

As per http://www.stats.uwo.ca/faculty/braun/ss3859/chapters/chapter_4/ch4.pdf (Slide 45)

So, I downloaded the training set of data from the supplementary data, and tried to make my own X matrix. However, I have tried inverting (X'X) in both excel and R without success. Excel gives me garbage output while R gives me a singularity error. However, if I eliminate the "column of one's" from the X matrix making it an k by X matrix, now (X'X) inverts just fine.

Just to see what is going on, I've also tried feeding the training set of points into Excel's Linnest function as well as R's lm , and I get a model out of both with co-efficient s that are very close to the published literature models. So this makes me think the matrix shouldn't have any issues like this.

Also note, the model has 5 quantitative and 1 qualitative dimension (with 42 possible results), so final model has 49 dimensions.

So I am wondering where I went wrong? why is my X'X matrix not invertible when I include the column of ones?



Assume $X$ is a $N \times M$ matrix. $X' X$ is singular if $X$'s rank is less than $\min(N, M)$: then resulting matrix will have rank less than $M$ and thus will be non-invertable.

Rank is sort of measure of how much unique data you have in the matrix. Why $X$ would have rank less than $M$ in the first place? One reason is feature correlations: if you have two identical columns, then you, essentially, have information about only one of them.

From geometrical perspective you can think of a matrix equation $A x = b$ where $A$ is a $N \times M$ matrix and $x \in \mathbb{R}^M$ and $b \in \mathbb{R}^N$ are vectors. Then each row of A defines a hyperplane in $\mathbb{R}^M$. Rank is number of independent hyperplanes. You need at least $M$ such (independent) hyperplanes to get a single point where they all intersect. If you have less independent hyperplanes, you'll get not a single point as a solution, but a whole hyperplane (line, if $rank(A) = M-1$ of them. If $A$ is square matrix, then solution is given using inverse of $A$: $x = A^{-1} b$. So if $A$ is of full-rank (that is, $rank(A)=M$), then solution is unique.

In you case $X' X$ (which is square matrix $M \times M$) will be of full-rank if and only if $rank(X) = M$

So, what can you do? There are 2 possibilities:

  1. Find and eliminate any duplicating columns in your data.
  2. Add a Tikhonov regularization (see Ridge Regression)
| cite | improve this answer | |
  • $\begingroup$ User "Barmaley.exe" was correct, I did have duplicate data, in one of the columns. The issue was the paper author kept including an intercept term in all math and calculations, which the author says has physical meaning. But after inspection has no mathematical meaning, as he then lowered the coef on the all the indicator variables by exactly the intercept, thus producing the exact same results as if there was no intercept or adjustment. $\endgroup$ – hswerdfe Dec 18 '14 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.