In linear regression, the parameters $b$ can be estimated this way (least squares):

\begin{align*} X'Xb &= X'y \\ b &= (X'X)^{-1}X'y \end{align*}

but from another point of view:

\begin{align*} X'Xb &= X'y \\ Xb &= (X')^{-1}X'y \\ &= Iy = y \end{align*}

This confuses me, because if this is true, then $e = y - Xb = 0$.

  • 3
    $\begingroup$ How do you define X' ? Existence of X' would imply that X' is square, which, unless I am mistaken is not the case. $\endgroup$ – Saulius Lukauskas Jan 5 '14 at 20:07
  • $\begingroup$ The expectation of the errors is zero. $\endgroup$ – D L Dahly Jan 5 '14 at 20:21

There is nothing wrong with your calculations, but they are special: For $X$ (and therefore $X'$) to be invertible, it must be a square matrix (and full rank of course). If it is a square matrix then it means that the number of regressors are exactly equal to the number of observations. Then you have an exactly identified system of linear equations, with number of equations (sample size), equal to the number of unknowns (number of coefficients), which, since the $X$ is invertible, has a unique solution, with zero residuals: $\mathbf y = \mathbf X\mathbf \beta \Rightarrow \mathbf X^{-1}\mathbf y=\mathbf \beta$. This result does not require that $X$ is the identity matrix.

| cite | improve this answer | |

Having your residuals sum to zero means that you have perfect linearity in your data, nothing wrong with that, but rare in most applications of least squares.

The problem is in


Because $X'$ isn't always invertible, or even square (a requirement for being invertible)

| cite | improve this answer | |
  • 2
    $\begingroup$ Nice, I think I got it now. $(X')^{-1}$ can exist when we use the maximal model, i.e. $X = I, b = y$, and in that case, the residuals are indeed all zeros. $\endgroup$ – qed Jan 5 '14 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.