10
$\begingroup$

Suppose

$$ \mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e} \, , \\ \mathbf{e} \sim \mathcal{N}(0,\mathbf{I}_P) \, . $$

We know that $\mathbf{\hat{b}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$ is the BLUE.

Is it also the UMVUE? I can only find a single source (page 6) that claims this, so I'm unsure.

In case $\mathbf{X}=\mathbf{1}$ it is true ($\mathbf{\hat{b}}$ becomes the sample mean).

But other, related results like Stein's example make me cautious.

And if it's true, then why isn't it more famous?

Improve this question
$\endgroup$
12
  • 2
    $\begingroup$ Another source: stat.wisc.edu/~doksum/STAT709/n709-36.pdf $\endgroup$
    – amoeba
    Commented May 15, 2018 at 12:48
  • 1
    $\begingroup$ You don't need to care about $l$, just look at the proof of (i) until the last line or so. They show that $\hat\beta$ is a function of complete sufficient statistic from which UMVUE follows by en.wikipedia.org/wiki/Lehmann%E2%80%93Scheff%C3%A9_theorem. Nothing is imposed here. Anyway, I just wanted to give this link. +1, good question. $\endgroup$
    – amoeba
    Commented May 15, 2018 at 13:10
  • 1
    $\begingroup$ stats.stackexchange.com/questions/288674/… $\endgroup$
    – Cagdas Ozgenc
    Commented May 15, 2018 at 13:23
  • 1
    $\begingroup$ Does this answer your question? Prove that OLS estimator of the intercept has minimum variance $\endgroup$
    – StubbornAtom
    Commented May 4, 2020 at 20:00
  • 1
    $\begingroup$ This is a great question and I am baffled that it is still not answered. I am in the same shoes. I found the bits and pieces that mention between the lines "if noise is Gaussian then it's also MVUE" but no more details. If anyone can give a concise answer between the relationship of OLS, BLUE and MVUE, that would be very helpful. $\endgroup$
    – divB
    Commented Oct 20, 2020 at 6:44

1 Answer 1

Reset to default
4
$\begingroup$

Under the assumptions $$ \begin{align} &\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e}, \;\mathbf X \;\text{full column rank},\\ &\mathbf e \mid \mathbf{X} \sim \mathop{\mathcal{N}}\left(\mathbf 0,\sigma^2\mathbf{I}\right),\;\sigma^2 \in \mathbb{R}_{>0} \end{align} $$ the OLS estimator $\hat{\mathbf{b}}=\left(\mathbf X^{\mathsf T}\mathbf X\right)^{-1}\mathbf X^{\mathsf T}\mathbf y$ is the UMVUE of $\mathbf b$.
This is clear from the facts that $\hat{\mathbf b}$ is unbiased and that $\mathop{\mathbb{V}}\left(\hat{\mathbf b}\right) = \sigma^2 \left(\mathbf X^{\mathsf T}\mathbf X\right)^{-1}$ is the inverse expected Fisher information of $\mathbf b$, i.e., $\hat{\mathbf b}$ attains the Cramér–Rao lower bound.
This result is in a sense more general than the Gauss–Markov theorem in that it's not restricted to linear estimators. On the other hand it's about linear regression with i.i.d. normal errors only.

Interestingly, under the more general assumptions $$ \begin{align} &\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e}, \;\mathbf X \;\text{full column rank},\\ &\mathbf e = \left(e_1,\ldots, e_n\right)^\mathsf{T},\\ &e_1,\ldots, e_n \mid \mathbf X \overset{\text{(c.)i.i.d.}}{\sim} \left(0, \sigma^2\right),\;\sigma^2 \in \mathbb{R}_{>0}, \end{align} $$ the OLS estimator $\hat{\mathbf{b}}=\left(\mathbf X^{\mathsf T}\mathbf X\right)^{-1}\mathbf X^{\mathsf T}\mathbf y$ is also the UMVUE of $\mathbf b$ among all estimators that are unbiased for all regression models that satisfy $$ \begin{align} &\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e}, \;\mathbf X \;\text{full column rank},\\ &\mathop{\mathbb{E}}\left(\mathbf e\mid \mathbf{X}\right)=\mathbf 0,\\ &\mathbf e = \left(e_1,\ldots, e_n\right)^\mathsf{T},\\ &e_1,\ldots, e_n \mid \mathbf X \;\text{(conditionally) independent},\\ &\mathop{\mathbb{V}}\left(\mathbf e \mid \mathbf{X}\right)=\mathop{\mathrm{diag}}\left(\sigma^2_1,\ldots,\sigma^2_n\right),\; \sigma^2_i \in \mathbb{R}_{>0},\\ \end{align} $$ i.e., for all linear regression models with independent and homo- or heteroscedastic errors (in particular, not only for the data-generating class of linear regression models with independent and homoscedastic errors).

References

Improve this answer
$\endgroup$
8
  • $\begingroup$ Why was it not answered for so long? +1. $\endgroup$
    – User1865345
    Commented Nov 20, 2022 at 3:30
  • $\begingroup$ Do you have any more context about this result? It seems crazy that such a basic fact was not established until this year (the proof doesn't look crazy either). $\endgroup$
    – John Madden
    Commented Nov 20, 2022 at 4:31
  • $\begingroup$ Yes. That was exactly my point @JohnMadden. But the outline looks neat and legit. So yeah. $\endgroup$
    – User1865345
    Commented Nov 20, 2022 at 7:16
  • $\begingroup$ @User1865345 Oh; from here it looked like you were asking why this question on SE took so long to be answered (which is also a reasonable question). $\endgroup$
    – John Madden
    Commented Nov 20, 2022 at 14:15
  • 1
    $\begingroup$ @JohnMadden After reading parts of the 2nd reference in my (edited) answer, in particular section 5 "The Results in Hansen (2022)" and double-checking with the Hansen paper I would say that the second part of my answer is not really a generalization of the Gauss–Markov theorem due to the independence assumption and requirement of unbiasedness over a class of distributions that is bigger than the assumed (smallest) class containing the DGP. $\endgroup$
    – statmerkur
    Commented Nov 20, 2022 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.