In what follows $y = (y_1,\dots,y_n)$ is a $n\times 1$ vector of random variables and $X = (x_{ij})$ is a $n\times d$ random matrix ($n>d$ tipically) with $\text{rank}(X)=d$ with probability 1.

Write $\beta := E(X'X)^{-1}E(X'y)$ and $u := y - X\beta$, and let $\hat{\beta} := (X'X)^{-1}X'y$ denote the OLS estimator. Also let $\tilde{\beta}=A'y$ denote some linear estimator of the form $A = \varphi\circ X$, for some $\varphi:M(n\times d)\to M(n\times d)$ (measurable), where $M(n\times d)$ is the space of $n\times d$ matrices.

The Gauss-Markov theorem states that, if

  1. $E(u |X) =0$ almost surely;
  2. $E(uu'|X) = \sigma^2 \mathrm{Id} $ for some $\sigma>0$ (where $\mathrm{Id}$ is the identity matrix);

then, whenever $\tilde{\beta}$ is unbiased for $\beta$, it holds that the matrix $$ E[(\tilde{\beta}-\beta)(\tilde{\beta}-\beta)'] - E[(\hat{\beta}-\beta)(\hat{\beta}-\beta)'] $$ is positive semi-definite. Now, in every textbook that I've come across, the unbiasedness assumption is invoked to conclude that $E(\tilde{\beta} | X) = \beta$ (almost surely) but this conclusion is strictly stronger than unbiasedness. Indeed, since $\tilde{\beta} = A'y = A'X\beta + A'u$, and since $A$ is $X$-measurable, we have by the assumption in item 1 above that $E(\tilde{\beta}|X) = A'X\beta$. At this points the canonical argument concludes that $\beta = A'X\beta$ and so on.

In my understanding, however, the definition of unbiasedness only allows me to conclude, using iterated expectations, that $\beta = E(A'X)\beta$, that is, $E(A'X) = \mathrm{Id}$. Am I missing something or is it implicit that the estimator $\tilde\beta$ is conditionally unbiased?

  • $\begingroup$ If I understand correctly, then, in the least square framework, $\tilde\beta$ being unbiased and $\tilde\beta$ being conditionally unbiased given $X$ is the same thing because $\tilde\beta$ can only exist when $X$ is given because it's a function of $X$. But I might not be understanding your question. $\endgroup$
    – mlofton
    Oct 25, 2020 at 7:08
  • $\begingroup$ I think this is not generally the case (maybe it's something particular to least squares) as the following (artificial) example shows: let $y_1,\dots,y_n$ be a random sample with $E(y_1) =: \beta$. Let, for the sake of argument, $x_i := y_i$, $i=1,\dots,n$. Put $\hat{\beta} = n^{-1}\sum_{i=1}^n x_i$ and $\tilde{\beta} = x_1$. Clearly $E(\hat{\beta}) = E(\tilde{\beta}) = \beta$ but $E(\tilde{\beta}\,|\, x_1,\dots, x_n) = x_1$. $\endgroup$
    – user127022
    Oct 25, 2020 at 12:22
  • $\begingroup$ I like your example and I think you're right. Forget my first comment. It was el-wrongo. I'll try to read-follow your answer later when I have more time. Hopefully, someone else can chime in because I think this is a very subtle topic-question. You're making me realize that maybe I don't understand Gauss Markov as well as I thought that I did !!!! $\endgroup$
    – mlofton
    Oct 26, 2020 at 14:24

1 Answer 1


In the paper The Gauss-Markov Theorem and Random Regressors, Juliet Popper Shaffer writes:

If attention is restricted to linear estimators ... that are conditionally unbiased, given $X$, the Gauss-Markov theorem applies. If, however, the estimator is required only to be unconditionally unbiased, the Gauss-Markov theorem may or may not hold, depending on what is known about the distribution of $X$.

Therefore, in the assumptions of the Gauss-Markov theorem with random $X$, it should be stated explicitly that $E(\tilde{\beta}\,|\,X) =\beta$.

There is an additional passage found in the “canonical proofs” that also bothers me, namely, that the equality $E(\tilde{\beta}\,|\,X) =\beta$ should hold for all $\beta\in\mathbb{R}^d$, as usually unbiasedness (conditional or unconditional) is introduced with a given fixed probability measure in mind. Since this post refers to the method of proof, I've written a statement which explicitly asserts every assumption that is used in these proofs:

Theorem Fix a measurable space $(\Omega,\mathscr{A})$, a random $n\times d$ matrix ${X}$ and a $n\times 1$ random vector $v$. Let $\mathfrak{M}$ denote the set of all probability measures $P$ satisfying the following

  1. $P(\text{rank}(X) = d) = 1$
  2. $E_P\big(v'v\big)<\infty$
  3. $E_P(v| X) = 0$
  4. $E_P( v v'| X) = \mathbf{Id}$

Let moreover $\psi:M(n\times d)\to M(n\times d)$ be measurable (where $M(n\times d)$ denotes the vector space of $n\times d$ matrices), and put $ X_\psi = \psi\circ X$.

Suppose that, for all $P \in\mathfrak{M}$, for all $\beta\in\mathbb R^{d}$ and all $\sigma>0$, it holds that $$ E _P ( X_\psi'( X\beta + \sigma v)\,|\, X) = \beta,\qquad\text{$P $-a.s.} $$ Then, for any $P \in\mathfrak M$, any $\beta\in\mathbb R^{d}$ and any $\sigma>0$ it holds that the $d\times d$ matrix $$ \text{var}_P ( X_\psi'( X\beta + \sigma v)\,|\, X) - \text{var}_P (( X' X)^{-1} X'( X \beta + \sigma v)\,|\, X) $$ is positive semidefinite, where $\text{var}_P $ is defined through $$ \text{var}_P ( z) := E _P ( z z') - E _P ( z)E _P ( z') $$ for all random vectors $ z$ such that $E _P ( z' z)<\infty$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.