In the linear regression model, the true error vector $U=Y-X\beta$ is based upon the true value of the unknown coefficient vector $\beta$. Meanwhile, the OLS residual vector $U^* =Y-X\beta^*$ uses the OLS estimator $\beta'$ of $\beta$, where $U^*$ and $\beta^*$ are the estimators of $U$ and $\beta$.

Then is it true that $U'U\lt {U^*}'U^*$?



Suppose you have a true relationship described by whatever slope $\beta\neq\beta^*$, but just two data points. You fit a simple linear regression. Then, the straight line will connect the two points and thus the residuals will be zero, and hence so will be ${U^*}'U^*$.

In general, recall that OLS is, by definition, the procedure that minimizes the sum of squared differences from $Y$, so that it cannot be beaten by other linear functions of the same set of regressors $X$.

In relation to this, the many posts on overfitting on this site and elsewhere provide further counterexamples: by making your model increasingly complex through, e.g., including powers of regressors, interactions etc., you can reduce the "training error" (that is, obtain smaller sum of squared residuals) of your model.

  • $\begingroup$ Thank you very much! Could you also take a look at stats.stackexchange.com/questions/243469/… $\endgroup$ – J.doe Nov 2 '16 at 6:06
  • $\begingroup$ This is a very special case and as such not really interesting from a practical perspective (but fine for a textbook exercise, of course). Is it the only possible example? $\endgroup$ – Richard Hardy Nov 2 '16 at 10:29
  • $\begingroup$ @RichardHardy, my impression was that the OP was asking for whether his conjecture is a general result, so that providing a counterexample that is as simple as possible would be helpful. I will try to make a more general statement. $\endgroup$ – Christoph Hanck Nov 2 '16 at 10:31
  • $\begingroup$ @ChristophHanck, the idea is not to burden you, and the OP has already accepted your answer, so it must be fine. I was just curious. $\endgroup$ – Richard Hardy Nov 2 '16 at 10:33

In the model (matrix notation)

$$y = X\beta + u$$

let the residual-maker matrix $M_X = I - X(X'X)^{-1}X' = I - P_X$, and note that $M_X X = 0$. Then

$$\hat u = M_Xy = M_Xu \implies \hat u'\hat u = u'M_Xu$$


$$\hat u'\hat u - u'u = u'(I - P_X)u - u'u = -u'P_Xu$$

$P_X$ is the Projection matrix, symmetric idempotent, and positive semi-definite, so

$$u'P_Xu \geq 0 \implies -u'P_Xu \leq 0$$.

This means that the sum of squared residuals are never larger than the sum of errors under the asumption of correct specification.

The issue of what happens when we include additional variables perhaps "irrelevant" to the regression, may be a different matter.


It's the other way around.

The simplest way to see this is that OLS estimate $\hat{\mathbf{b}}$ is chosen to minimize $\left(\mathbf{y} - X\mathbf{b}\right)'\left(\mathbf{y} - X \mathbf{b}\right)$. Therefore $\left(\mathbf{y} - X\mathbf{b}\right)'\left(\mathbf{y} - X \mathbf{b}\right) \geq \left(\mathbf{y} - X\hat{\mathbf{b}}\right)'\left(\mathbf{y} - X \hat{\mathbf{b}}\right)$ for any $\mathbf{b}$, including the true value $\boldsymbol{\beta}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.