Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In linear regression, the parameters $b$ can be estimated this way (least squares):

\begin{align*} X'Xb &= X'y \\ b &= (X'X)^{-1}X'y \end{align*}

but from another point of view:

\begin{align*} X'Xb &= X'y \\ Xb &= (X')^{-1}X'y \\ &= Iy = y \end{align*}

This confuses me, because if this is true, then $e = y - Xb = 0$.

share|improve this question
3  
How do you define X' ? Existence of X' would imply that X' is square, which, unless I am mistaken is not the case. –  Saulius Lukauskas Jan 5 at 20:07
    
The expectation of the errors is zero. –  D L Dahly Jan 5 at 20:21

2 Answers 2

up vote 5 down vote accepted

There is nothing wrong with your calculations, but they are special: For $X$ (and therefore $X'$) to be invertible, it must be a square matrix (and full rank of course). If it is a square matrix then it means that the number of regressors are exactly equal to the number of observations. Then you have an exactly identified system of linear equations, with number of equations (sample size), equal to the number of unknowns (number of coefficients), which, since the $X$ is invertible, has a unique solution, with zero residuals: $\mathbf y = \mathbf X\mathbf \beta \Rightarrow \mathbf X^{-1}\mathbf y=\mathbf \beta$. This result does not require that $X$ is the identity matrix.

share|improve this answer

Having your residuals sum to zero means that you have perfect linearity in your data, nothing wrong with that, but rare in most applications of least squares.

The problem is in

$Xb=(X')^{-1}X'y$

Because $X'$ isn't always invertible, or even square (a requirement for being invertible)

share|improve this answer
2  
Nice, I think I got it now. $(X')^{-1}$ can exist when we use the maximal model, i.e. $X = I, b = y$, and in that case, the residuals are indeed all zeros. –  qed Jan 5 at 20:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.