I'm trying to follow Faraway's proof of the Gauss-Markov model in his book Linear Models with R, 2nd Ed., on pages 22-23, but I have been having numerous difficulties. The latest difficulty is thick in the weeds, so I will type out his proof exactly.

Suppose $E\varepsilon=0$ and $\operatorname{var}\varepsilon=\sigma^2I.$ Suppose also that the structural part of the model, $EY=X\beta$ is correct. (Clearly these are big assumptions and so we will address the implications of this later.) Let $\psi=c^T\beta$ be an estimable function; then the Gauss-Markov theorem states that in the class of all unbiased linear estimates of $\psi,\; \hat\psi=c^T\hat\beta$ has the minimum variance and is unique.

We prove this theorem. Suppose $a^Ty$ is some unbiased estimate of $c^T\beta$ so that: \begin{align*} Ea^Ty&=c^T\beta\qquad\forall\beta\\ a^TX\beta&=c^T\beta\qquad\forall\beta \end{align*} which means that $a^TX=c^T.$ This implies that $c$ must be in the range space of $X^T$ [$X^Ta=c$, ACK] which in turn implies that $c$ is also in the range space of $X^TX$ which means there exists a $\lambda$ such that $c=X^TX\lambda$ so: $$c^T\hat\beta=\lambda^TX^TX\hat\beta=\lambda^TX^Ty$$ Now we can show that the least squares estimator has the minimum variance - pick an arbitrary estimate $a^Ty$ and compute its variance: \begin{align*} \operatorname{var}\!\left(a^Ty\right) &=\operatorname{var}\!\left(a^Ty-c^T\hat\beta+c^T\hat\beta\right)\\ &=\operatorname{var}\!\left(a^Ty-\lambda^TX^Ty+c^T\hat\beta\right)\\ &=\operatorname{var}\!\left(a^Ty-\lambda^TX^Ty\right) +\operatorname{var}\!\left(c^T\hat\beta\right) +2\operatorname{cov}\!\left(a^Ty-\lambda^TX^Ty,\lambda^TX^Ty\right) \end{align*} but $$\operatorname{cov}\!\left(a^Ty-\lambda^TX^Ty,\lambda^TX^Ty\right) =\left(a^T-\lambda^TX^T\right)\sigma^2IX\lambda \\ \dots$$

My Question: The rest of the proof is very straight-forward, but: where did that last line come from? I'm just astonished at how many skipped steps that has to represent, and I am rather of the opinion that the rest of the steps in that section are trivial compared with this "and-then-a-miracle-occurred" kind of step.

As a general comment: I have searched through many proofs of the Gauss-Markov theorem. Why are the proofs so universally opaque? Is there a step-by-step proof of the Gauss-Markov theorem that does not assume too much of the reader?


1 Answer 1


They are omitting some algebraic manipulations which use some properties of covariance described here (this is the citation linked on wikipedia for these properties):


$$\mathrm{cov}(a^Ty - \lambda^TX^Ty, \lambda^TX^Ty) = (a^T- \lambda^TX^T)\mathrm{cov}(y, \lambda^TX^Ty).$$

Here we are pulling out a constant factor from the left argument of $\mathrm{cov}$.

$$ = (a^T - \lambda^TX^T)\mathrm{cov}(y, y)X\lambda$$

Here we are pulling out a constant factor from the right argument of $\mathrm{cov}$ (but the right argument must be transposed when we do so).

Now we are left with the covariance matrix of the vector $y$. By assumption, $y_i$ and $y_j$ are uncorrelated for $i \neq j$ (this is independence of observations). So this is a diagonal matrix. And again by assumption, $\mathrm{var}(y_i) = \mathrm{var}(y_j) = \sigma^2$ for all $i, j$. This is homoskedasticity. So finally, you are left with the equality in the proof.

On the opacity of the proof: I think it is just about familiarity with properties of covariance and matrix algebra. I also found the Gauss Markov proof really hard to follow (the proof in Elements of Statistical Learning is not easier to read). But once you see it and understand the rules for covariance of linear transformations, I think it becomes easy to see why an author might omit those steps.

For many people, regression models are now their first introduction to statistics beyond combinatorics and simple probability theory, so it's natural to dive into a proof like this. But many texts are written assuming the reader is following a statistics curriculum which includes, I assume, a prerequisite course where you prove these kinds of properties and drill them in homework exercises. It may not have occurred to authors that there are people doing things "out of order" for whom the Gauss Markov theorem is very interesting, but who haven't become very familiar with these sorts of manipulations.

  • 1
    $\begingroup$ Well, I did just work through a calculus-based mathematical statistics textbook (read every word, worked all 1k+ problems), but it was a slightly older one and didn't use matrices a whole lot. It gave many covariance properties, but not in the matrix setting, and certainly not that one. Thanks for pointing it out! That certainly clears up the whole thing. $\endgroup$ Jan 14, 2022 at 1:26
  • $\begingroup$ Sorry, I didn't mean to imply you hadn't done the work. I struggled with the same part of the G-M proof, and when I finally figured it out I had the strong impression that it was something I "should have" known. But maybe I was wrong and the proofs are just written very densely! $\endgroup$ Jan 14, 2022 at 1:33
  • $\begingroup$ No offense taken, I assure you! It's all a matter of what an author can reasonably expect readers to know. If all the up-to-date mathematical statistics textbooks take a heavily matrix-based approach and they all teach that covariance fact, then it's definitely one step in a proof and quite straight-forward to follow. But if, like me, it's simply not a known fact, then it is bewildering. Fact of life. $\endgroup$ Jan 14, 2022 at 3:40
  • 1
    $\begingroup$ You can derive basic variance / covariance facts yourself from a very basic perspective of considering the elements term by term, you should be able to rapidly convince yourself how things should work. It's not some big mystery. (You don't have to approach it that way, but it can help if the linear algebra is not super obvious right from the get go - which it might not be, since it isn't for many people at first.) The first time you encounter these things it can help clarify just how it all works. $\endgroup$
    – Glen_b
    Jan 14, 2022 at 3:44
  • $\begingroup$ this was all quite interesting and educational. thanks. my guess-suggestion is to take a look at CR Rao's text. amazon.com/Linear-Statistical-Inference-Its-Applications/dp/… I can't be sure that it's in there in some nice way but of the statistical texts that I'm familiar with, that would be the one I would put my money on. $\endgroup$
    – mlofton
    Jan 14, 2022 at 4:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.