# Using the unbiasedness assumption in the proof of the Gauss-Markov Theorem

In what follows $$y = (y_1,\dots,y_n)$$ is a $$n\times 1$$ vector of random variables and $$X = (x_{ij})$$ is a $$n\times d$$ random matrix ($$n>d$$ tipically) with $$\text{rank}(X)=d$$ with probability 1.

Write $$\beta := E(X'X)^{-1}E(X'y)$$ and $$u := y - X\beta$$, and let $$\hat{\beta} := (X'X)^{-1}X'y$$ denote the OLS estimator. Also let $$\tilde{\beta}=A'y$$ denote some linear estimator of the form $$A = \varphi\circ X$$, for some $$\varphi:M(n\times d)\to M(n\times d)$$ (measurable), where $$M(n\times d)$$ is the space of $$n\times d$$ matrices.

The Gauss-Markov theorem states that, if

1. $$E(u |X) =0$$ almost surely;
2. $$E(uu'|X) = \sigma^2 \mathrm{Id}$$ for some $$\sigma>0$$ (where $$\mathrm{Id}$$ is the identity matrix);

then, whenever $$\tilde{\beta}$$ is unbiased for $$\beta$$, it holds that the matrix $$E[(\tilde{\beta}-\beta)(\tilde{\beta}-\beta)'] - E[(\hat{\beta}-\beta)(\hat{\beta}-\beta)']$$ is positive semi-definite. Now, in every textbook that I've come across, the unbiasedness assumption is invoked to conclude that $$E(\tilde{\beta} | X) = \beta$$ (almost surely) but this conclusion is strictly stronger than unbiasedness. Indeed, since $$\tilde{\beta} = A'y = A'X\beta + A'u$$, and since $$A$$ is $$X$$-measurable, we have by the assumption in item 1 above that $$E(\tilde{\beta}|X) = A'X\beta$$. At this points the canonical argument concludes that $$\beta = A'X\beta$$ and so on.

In my understanding, however, the definition of unbiasedness only allows me to conclude, using iterated expectations, that $$\beta = E(A'X)\beta$$, that is, $$E(A'X) = \mathrm{Id}$$. Am I missing something or is it implicit that the estimator $$\tilde\beta$$ is conditionally unbiased?

• If I understand correctly, then, in the least square framework, $\tilde\beta$ being unbiased and $\tilde\beta$ being conditionally unbiased given $X$ is the same thing because $\tilde\beta$ can only exist when $X$ is given because it's a function of $X$. But I might not be understanding your question. Oct 25, 2020 at 7:08
• I think this is not generally the case (maybe it's something particular to least squares) as the following (artificial) example shows: let $y_1,\dots,y_n$ be a random sample with $E(y_1) =: \beta$. Let, for the sake of argument, $x_i := y_i$, $i=1,\dots,n$. Put $\hat{\beta} = n^{-1}\sum_{i=1}^n x_i$ and $\tilde{\beta} = x_1$. Clearly $E(\hat{\beta}) = E(\tilde{\beta}) = \beta$ but $E(\tilde{\beta}\,|\, x_1,\dots, x_n) = x_1$. Oct 25, 2020 at 12:22
• I like your example and I think you're right. Forget my first comment. It was el-wrongo. I'll try to read-follow your answer later when I have more time. Hopefully, someone else can chime in because I think this is a very subtle topic-question. You're making me realize that maybe I don't understand Gauss Markov as well as I thought that I did !!!! Oct 26, 2020 at 14:24

In the paper The Gauss-Markov Theorem and Random Regressors, Juliet Popper Shaffer writes:

If attention is restricted to linear estimators ... that are conditionally unbiased, given $$X$$, the Gauss-Markov theorem applies. If, however, the estimator is required only to be unconditionally unbiased, the Gauss-Markov theorem may or may not hold, depending on what is known about the distribution of $$X$$.

Therefore, in the assumptions of the Gauss-Markov theorem with random $$X$$, it should be stated explicitly that $$E(\tilde{\beta}\,|\,X) =\beta$$.

There is an additional passage found in the “canonical proofs” that also bothers me, namely, that the equality $$E(\tilde{\beta}\,|\,X) =\beta$$ should hold for all $$\beta\in\mathbb{R}^d$$, as usually unbiasedness (conditional or unconditional) is introduced with a given fixed probability measure in mind. Since this post refers to the method of proof, I've written a statement which explicitly asserts every assumption that is used in these proofs:

Theorem Fix a measurable space $$(\Omega,\mathscr{A})$$, a random $$n\times d$$ matrix $${X}$$ and a $$n\times 1$$ random vector $$v$$. Let $$\mathfrak{M}$$ denote the set of all probability measures $$P$$ satisfying the following

1. $$P(\text{rank}(X) = d) = 1$$
2. $$E_P\big(v'v\big)<\infty$$
3. $$E_P(v| X) = 0$$
4. $$E_P( v v'| X) = \mathbf{Id}$$

Let moreover $$\psi:M(n\times d)\to M(n\times d)$$ be measurable (where $$M(n\times d)$$ denotes the vector space of $$n\times d$$ matrices), and put $$X_\psi = \psi\circ X$$.

Suppose that, for all $$P \in\mathfrak{M}$$, for all $$\beta\in\mathbb R^{d}$$ and all $$\sigma>0$$, it holds that $$E _P ( X_\psi'( X\beta + \sigma v)\,|\, X) = \beta,\qquad\text{P -a.s.}$$ Then, for any $$P \in\mathfrak M$$, any $$\beta\in\mathbb R^{d}$$ and any $$\sigma>0$$ it holds that the $$d\times d$$ matrix $$\text{var}_P ( X_\psi'( X\beta + \sigma v)\,|\, X) - \text{var}_P (( X' X)^{-1} X'( X \beta + \sigma v)\,|\, X)$$ is positive semidefinite, where $$\text{var}_P$$ is defined through $$\text{var}_P ( z) := E _P ( z z') - E _P ( z)E _P ( z')$$ for all random vectors $$z$$ such that $$E _P ( z' z)<\infty$$.