0
$\begingroup$

I was going through this answer which gives total sum of squares as

$TSS=\sum_i(Y_i-\hat{Y}_i)^2+\sum_i(\hat{Y}_i-\bar{Y})^2+2\sum_i(Y_i-\hat{Y}_i)(\hat{Y}_i-\bar{Y})$

It then says:

In a model with an intercept, it can be shown that the third term on the right hand side is zero, but this can not be shown for a model without an intercept (see e.g. D.N. Gujaratti, "Basic Econometrics" or W.H. Greene, "Econometric Analysis").

I am not able to find the proof of above quoted fact in Gujarati's book, nor am able to guess it my self. Can someone please give me proof and if possible some visual intuition of it?

(PS: am quite noob to stats)

$\endgroup$
6
  • $\begingroup$ All it takes to prove such a negative statement is a counterexample. Contemplate the $(x,y)$ dataset $(1,0),$ $(1,2).$ For visual intuition, plot the points and plot the (obvious) fits with and without intercepts. $\endgroup$
    – whuber
    Jul 15, 2021 at 21:42
  • $\begingroup$ Does "without an intercept" mean "with the intercept constrained to equal 0" (or some other constant value)? $\endgroup$
    – Alexis
    Jul 15, 2021 at 22:31
  • 2
    $\begingroup$ @Alexis "Without an intercept" means literally that: there is no explicit constant term in the model. That's equivalent to including an intercept and constraining it to zero, of course. A synonymous phrase is "regression through the origin." $\endgroup$
    – whuber
    Jul 15, 2021 at 22:50
  • $\begingroup$ Thanks @whuber that was my suspicion, but was not sure if it sometimes had some other meaning. $\endgroup$
    – Alexis
    Jul 15, 2021 at 23:12
  • $\begingroup$ @whuber please help me out or at least point me to some link / source explaining intuition. Am unable to imagine as you said in first comment. $\endgroup$
    – Rnj
    Jul 16, 2021 at 8:59

1 Answer 1

1
$\begingroup$

Are you familiar with linear algebra?

Let $Y=X\beta+\epsilon$. $\hat\beta=(X^TX)^{-1}X^TY$ and $\hat Y=X\hat\beta=PY$, where $P=X(X^TX)^{-1}X^T$. The $P$ matrix is symmetric and idempotent, it is the projection matrix of $Y$ onto $\mathcal{C}(X)$, the column space of $X$ (see Wikipedia).

Let $1$ be a column vector of ones and $J=11^T$ a square matrix of ones. You have: $$1^TY=\sum_iY_i,\qquad JY=\begin{bmatrix} \sum_iY_i \\ \vdots \\ \sum_iY_i \end{bmatrix}, \qquad \frac1n JY=\begin{bmatrix}\frac1n\sum_iY_i\\ \vdots \\ \frac1n\sum_iY_i\end{bmatrix}=\begin{bmatrix}\bar Y\\ \vdots \\ \bar Y\end{bmatrix}$$ The $n^{-1}J$ matrix is symmetric and idempotent.

In matrix form, $$\begin{align*}\sum_i(Y_i-\hat Y_i)^2&=(Y-PY)^T(Y-PY)\\ \sum_i(\hat Y_i-\bar Y)^2&=(PY-n^{-1}JY)^T(PY-n^{-1}JY)\\ 2\sum_i(Y_i-\hat Y_i)(\hat Y_i-\bar Y)&=2(Y-PY)^T(PY-n^{-1}JY) \\ &=2Y^T(I-P)^T(P-n^{-1}J)Y\end{align*}$$ where $I$ is the identity matrix.

$(I-P)^T(P-n^{-1}J)=P-n^{-1}J-P+Pn^{-1}J=Pn^{-1}J-n^{-1}J$. But $P$ is the projection matrix onto $\mathcal{C}(X)$, so if there is a column of ones in $X$, then each column of $J$ is in $\mathcal{C}(X)$, $PJ=J$ (each column in $J$ is projected onto itself) and $Pn^{-1}J=n^{-1}PJ=n^{-1}J$.

Therefore, $Pn^{-1}J-n^{-1}J=n^{-1}J-n^{-1}J=0$.

This is why "In a model with an intercept, it can be shown that the third term on the right hand side is zero." As to "this can not be shown for a model without an intercept" (notice that "this can not be shown" is not "this can not happen"), the third term could be zero even in a model without an intercept, but only if the columns in $X$ generate a vector of ones.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.