10
$\begingroup$

Suppose

$$ \mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e} \, , \\ \mathbf{e} \sim \mathcal{N}(0,\mathbf{I}_P) \, . $$

We know that $\mathbf{\hat{b}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$ is the BLUE.

Is it also the UMVUE? I can only find a single source (page 6) that claims this, so I'm unsure.

In case $\mathbf{X}=\mathbf{1}$ it is true ($\mathbf{\hat{b}}$ becomes the sample mean).

But other, related results like Stein's example make me cautious.

And if it's true, then why isn't it more famous?

$\endgroup$
12
  • 2
    $\begingroup$ Another source: stat.wisc.edu/~doksum/STAT709/n709-36.pdf $\endgroup$
    – amoeba
    Commented May 15, 2018 at 12:48
  • 1
    $\begingroup$ You don't need to care about $l$, just look at the proof of (i) until the last line or so. They show that $\hat\beta$ is a function of complete sufficient statistic from which UMVUE follows by en.wikipedia.org/wiki/Lehmann%E2%80%93Scheff%C3%A9_theorem. Nothing is imposed here. Anyway, I just wanted to give this link. +1, good question. $\endgroup$
    – amoeba
    Commented May 15, 2018 at 13:10
  • 1
    $\begingroup$ stats.stackexchange.com/questions/288674/… $\endgroup$ Commented May 15, 2018 at 13:23
  • 1
    $\begingroup$ Does this answer your question? Prove that OLS estimator of the intercept has minimum variance $\endgroup$ Commented May 4, 2020 at 20:00
  • 1
    $\begingroup$ This is a great question and I am baffled that it is still not answered. I am in the same shoes. I found the bits and pieces that mention between the lines "if noise is Gaussian then it's also MVUE" but no more details. If anyone can give a concise answer between the relationship of OLS, BLUE and MVUE, that would be very helpful. $\endgroup$
    – divB
    Commented Oct 20, 2020 at 6:44

1 Answer 1

4
$\begingroup$

Under the assumptions $$ \begin{align} &\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e}, \;\mathbf X \;\text{full column rank},\\ &\mathbf e \mid \mathbf{X} \sim \mathop{\mathcal{N}}\left(\mathbf 0,\sigma^2\mathbf{I}\right),\;\sigma^2 \in \mathbb{R}_{>0} \end{align} $$ the OLS estimator $\hat{\mathbf{b}}=\left(\mathbf X^{\mathsf T}\mathbf X\right)^{-1}\mathbf X^{\mathsf T}\mathbf y$ is the UMVUE of $\mathbf b$.
This is clear from the facts that $\hat{\mathbf b}$ is unbiased and that $\mathop{\mathbb{V}}\left(\hat{\mathbf b}\right) = \sigma^2 \left(\mathbf X^{\mathsf T}\mathbf X\right)^{-1}$ is the inverse expected Fisher information of $\mathbf b$, i.e., $\hat{\mathbf b}$ attains the Cramér–Rao lower bound.
This result is in a sense more general than the Gauss–Markov theorem in that it's not restricted to linear estimators. On the other hand it's about linear regression with i.i.d. normal errors only.

Interestingly, under the more general assumptions $$ \begin{align} &\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e}, \;\mathbf X \;\text{full column rank},\\ &\mathbf e = \left(e_1,\ldots, e_n\right)^\mathsf{T},\\ &e_1,\ldots, e_n \mid \mathbf X \overset{\text{(c.)i.i.d.}}{\sim} \left(0, \sigma^2\right),\;\sigma^2 \in \mathbb{R}_{>0}, \end{align} $$ the OLS estimator $\hat{\mathbf{b}}=\left(\mathbf X^{\mathsf T}\mathbf X\right)^{-1}\mathbf X^{\mathsf T}\mathbf y$ is also the UMVUE of $\mathbf b$ if $\hat{\mathbf{b}}$ is unbiased for all regression models that satisfy $$ \begin{align} &\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e}, \;\mathbf X \;\text{full column rank},\\ &\mathop{\mathbb{E}}\left(\mathbf e\mid \mathbf{X}\right)=\mathbf 0,\\ &\mathbf e = \left(e_1,\ldots, e_n\right)^\mathsf{T},\\ &e_1,\ldots, e_n \mid \mathbf X \;\text{(conditionally) independent},\\ &\mathop{\mathbb{V}}\left(\mathbf e \mid \mathbf{X}\right)=\mathop{\mathrm{diag}}\left(\sigma^2_1,\ldots,\sigma^2_n\right),\; \sigma^2_i \in \mathbb{R}_{>0},\\ \end{align} $$ i.e., for all linear regression models with independent and homo- or heteroscedastic errors (in particular, not only for the data-generating class of linear regression models with independent and homoscedastic errors).


References

$\endgroup$
8
  • $\begingroup$ Why was it not answered for so long? +1. $\endgroup$ Commented Nov 20, 2022 at 3:30
  • $\begingroup$ Do you have any more context about this result? It seems crazy that such a basic fact was not established until this year (the proof doesn't look crazy either). $\endgroup$ Commented Nov 20, 2022 at 4:31
  • $\begingroup$ Yes. That was exactly my point @JohnMadden. But the outline looks neat and legit. So yeah. $\endgroup$ Commented Nov 20, 2022 at 7:16
  • $\begingroup$ @User1865345 Oh; from here it looked like you were asking why this question on SE took so long to be answered (which is also a reasonable question). $\endgroup$ Commented Nov 20, 2022 at 14:15
  • 1
    $\begingroup$ @JohnMadden After reading parts of the 2nd reference in my (edited) answer, in particular section 5 "The Results in Hansen (2022)" and double-checking with the Hansen paper I would say that the second part of my answer is not really a generalization of the Gauss–Markov theorem due to the independence assumption and requirement of unbiasedness over a class of distributions that is bigger than the assumed (smallest) class containing the DGP. $\endgroup$
    – statmerkur
    Commented Nov 20, 2022 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.