I a Multiple linear regression model, from published literature, I am implementing a spreadsheet to generate new predictions based on the published model. the literature stated Coefficients and the error in each as well as the RMS error. However to estimate prediction interval i need to know the matrix (X'X)^-1 Where X is a k+1 by N model matrix. and X' it the transpose of X

As per http://www.stats.uwo.ca/faculty/braun/ss3859/chapters/chapter_4/ch4.pdf (Slide 45)

So, I downloaded the training set of data from the supplementary data, and tried to make my own X matrix. However, I have tried inverting (X'X) in both excel and R without success. Excel gives me garbage output while R gives me a singularity error. However, if I eliminate the "column of one's" from the X matrix making it an k by X matrix, now (X'X) inverts just fine.

Just to see what is going on, I've also tried feeding the training set of points into Excel's Linnest function as well as R's lm , and I get a model out of both with co-efficient s that are very close to the published literature models. So this makes me think the matrix shouldn't have any issues like this.

Also note, the model has 5 quantitative and 1 qualitative dimension (with 42 possible results), so final model has 49 dimensions.

So I am wondering where I went wrong? why is my X'X matrix not invertible when I include the column of ones?


  • 2
    $\begingroup$ It is not an answer but rather a comment but check this post: johndcook.com/blog/2010/01/19/dont-invert-that-matrix $\endgroup$ – Tim Dec 17 '14 at 20:12
  • $\begingroup$ For related information, please search our site. And please make sure to look at stats.stackexchange.com/questions/3392: this is one case where Excel is definitely not the tool to use. $\endgroup$ – whuber Dec 17 '14 at 20:33
  • 1
    $\begingroup$ If you include all levels of a factor, the sum of its dummies will be 1, so you'll have perfect multicollinearity with the intercept. Usually one is omitted. $\endgroup$ – Glen_b Dec 17 '14 at 23:04

Assume $X$ is a $N \times M$ matrix. $X' X$ is singular if $X$'s rank is less than $\min(N, M)$: then resulting matrix will have rank less than $M$ and thus will be non-invertable.

Rank is sort of measure of how much unique data you have in the matrix. Why $X$ would have rank less than $M$ in the first place? One reason is feature correlations: if you have two identical columns, then you, essentially, have information about only one of them.

From geometrical perspective you can think of a matrix equation $A x = b$ where $A$ is a $N \times M$ matrix and $x \in \mathbb{R}^M$ and $b \in \mathbb{R}^N$ are vectors. Then each row of A defines a hyperplane in $\mathbb{R}^M$. Rank is number of independent hyperplanes. You need at least $M$ such (independent) hyperplanes to get a single point where they all intersect. If you have less independent hyperplanes, you'll get not a single point as a solution, but a whole hyperplane (line, if $rank(A) = M-1$ of them. If $A$ is square matrix, then solution is given using inverse of $A$: $x = A^{-1} b$. So if $A$ is of full-rank (that is, $rank(A)=M$), then solution is unique.

In you case $X' X$ (which is square matrix $M \times M$) will be of full-rank if and only if $rank(X) = M$

So, what can you do? There are 2 possibilities:

  1. Find and eliminate any duplicating columns in your data.
  2. Add a Tikhonov regularization (see Ridge Regression)
  • $\begingroup$ User "Barmaley.exe" was correct, I did have duplicate data, in one of the columns. The issue was the paper author kept including an intercept term in all math and calculations, which the author says has physical meaning. But after inspection has no mathematical meaning, as he then lowered the coef on the all the indicator variables by exactly the intercept, thus producing the exact same results as if there was no intercept or adjustment. $\endgroup$ – hswerdfe Dec 18 '14 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.