9
$\begingroup$

I know that OLS is unbiased but not efficient under heteroscedasticity in a linear regression setting.

In Wikipedia

http://en.wikipedia.org/wiki/Minimum_mean_square_error

The MMSE estimator is asymptotically unbiased and it converges in distribution to the normal distribution: $\sqrt{n}(\hat{x} - x) \xrightarrow{d} \mathcal{N}\left(0 , I^{-1}(x)\right)$, where I(x) is the Fisher information of x. Thus, the MMSE estimator is asymptotically efficient.

MMSE is claimed to be asymptotically efficient. I am a little confused here.

Does this mean OLS is not efficient in finite sample, but efficient asymptotically under heteroscedasticity?

Critique of the current answers: So far the proposed answers don't address the limiting distribution.

Thanks in advance

$\endgroup$
  • $\begingroup$ That is quite a long wikipedia article. Since, in addition, these are subject to change, would you mind citing the passage causing confusion? $\endgroup$ – hejseb Apr 1 '15 at 13:45
  • 1
    $\begingroup$ The Fisher information is derived from the likelihood function. So it's implicitly implying that the likelihood was specified correctly. i.e. The statement your referring to assumes, if there is any heteroscedasticity, the regression was weighted in a way such that heteroscedasticity was correctly specified. See en.wikipedia.org/wiki/Least_squares#Weighted_least_squares. In practice we often do not know the form of heteroscedasticity, so we sometimes accept the inefficiency rather than take the chance of biasing the regression by miss specifying weighting schemes. $\endgroup$ – Zachary Blumenfeld Apr 1 '15 at 16:51
  • $\begingroup$ @ZacharyBlumenfeld There was no assumption on the distribution of x in the article. How did we end up with the Fisher information? $\endgroup$ – Cagdas Ozgenc Apr 1 '15 at 17:22
  • 1
    $\begingroup$ See en.wikipedia.org/wiki/Fisher_information The article implies a distribution on $x$ and $e$ when it takes expectations in the definition section. Note that homoscedasticity was never assumed there. In the context of OLS, homoscedacticity assumed $\mathbf{e} \sim N(0,\sigma I)$, $I$ the identity matrix. Heteroscedacticity allows for $\mathbf{e} \sim N(0,D)$, any $D$ diagonal Positive semi-definite. Using $D$ would result in a different Fisher information than would using $\sigma I$. $\endgroup$ – Zachary Blumenfeld Apr 1 '15 at 17:44
  • $\begingroup$ where can I see a proof of this fact that "MMSE converges in distribution to the normal distribution?" $\endgroup$ – Hajir Apr 5 '18 at 21:58
3
$\begingroup$

The article never assumed homoskadasticity in the definition. To put it in the context of the article, homoskedasticity would be saying $$ E\{(\hat x-x)(\hat x-x)^T\}=\sigma I $$ Where $I$ is the $n\times n$ identity matrix and $\sigma$ is a scalar positive number. Heteroscadasticity allows for

$$ E\{(\hat x-x)(\hat x-x)^T\}=D $$

Any $D$ diaganol positive definite. The article defines the covariance matrix in the most general way possible, as the centered second moment of some implicit multi-variate distribution. we must know the multivariate distribution of $e$ to obtain an asymptotically efficient and consistent estimate of $\hat x$. This will come from a likelihood function (which is a mandatory component of the posterior). For example, assume $e \sim N(0,\Sigma)$ (i.e $E\{(\hat x-x)(\hat x-x)^T\}=\Sigma$. Then the implied likelihood function is $$ \log[L]=\log[\phi(\hat x -x, \Sigma)] $$ Where $\phi$ is the multivariate normal pdf.

The fisher information matrix may be written as $$ I(x)=E\bigg[\bigg(\frac{\partial}{\partial x}\log[L]\bigg)^2 \,\bigg|\,x \bigg] $$ see en.wikipedia.org/wiki/Fisher_information for more. It is from here that we can derive $$ \sqrt{n}(\hat x -x) \rightarrow^d N(0, I^{-1}(x)) $$ The above is using a quadratic loss function but does not assuming homoscedasticity.

In the context of OLS, where we regress $y$ on $x$ we assume $$ E\{y|x\}=x'\beta $$ The likelihood implied is $$ \log[L]=\log[\phi(y-x'\beta, \sigma I)] $$ Which may be conveniently rewritten as $$ \log[L]=\sum_{i=1}^n\log[\varphi(y-x'\beta, \sigma)] $$ $\varphi$ the univariate normal pdf. The fisher information is then $$ I(\beta)=[\sigma (xx')^{-1}]^{-1} $$

If homoskedasticity is not meet, then the Fisher information as stated is miss specified (but the conditional expectation function is still correct) so the estimates of $\beta$ will be consistent but inefficient. We could rewrite the likelihood to account for heteroskacticity and the regression is efficient i.e. we can write $$ \log[L]=\log[\phi(y-x'\beta, D)] $$ This is equivalent to certain forms of Generalized Least Squares, such as Weighted least squares. However, this will change the Fisher information matrix. In practice we often don't know the form of heteroscedasticity so we sometimes prefer accept the inefficiency rather than chance biasing the regression by miss specifying weighting schemes. In such cases the asymptotic covariance of $\beta$ is not $\frac{1}{n}I^{-1}(\beta)$ as specified above.

$\endgroup$
  • $\begingroup$ Thanks for all the time you spent. However I think that wiki entry is total crap. MMSE will not give efficiency, and nowhere it is specified that the samples are weighted appropriately. Moreover even if we assume that the samples are weighted, it still not an efficient estimator unless the distribution is Gaussian, which is also not specified. $\endgroup$ – Cagdas Ozgenc Apr 1 '15 at 19:19
  • $\begingroup$ @CagdasOzgenc I respectfully disagree. The article is phrased in a general Bayesian way which may include regression, but also many other models (it seems to be aimed more toward Kalman filter). The likelihood is the most efficient estimator when it is known, this is a basic property of likelihood. What your saying applies strictly to a subset of regression models (albeit among the most widely applied models) where normality is assumed when deriving first order conditions. $\endgroup$ – Zachary Blumenfeld Apr 1 '15 at 20:00
  • 1
    $\begingroup$ You said it yourself. Unfortunately the article is not about likelihood estimator. It is the Minimum Mean Square Estimator, which is efficient when certain conditions are satisfied. $\endgroup$ – Cagdas Ozgenc Apr 1 '15 at 20:39
  • $\begingroup$ Alright I agree to disagree :) Perhaps there is a conflict with the definition of MMSE between how it's used in frequentest regression and how it's applied here in a more Bayesian setting. Perhaps they should invent a new name for it. Nevertheless likelihoods (or perhaps other non-parametric estimations) are implied when taking independent expectations over every single squared residual. especially in a Bayesian setting (otherwise how would we estimate it?). After Googling I found a lot of similar results to the one on Wikipedia. Anyway I agree that terminology is being abused. $\endgroup$ – Zachary Blumenfeld Apr 1 '15 at 22:15
2
$\begingroup$

No, OLS is not efficient under heteroscedasticity. Efficiency of an estimator is obtained if the estimator has the least variance among other possible estimators. Statements about efficiency in OLS are made regardless of the limiting distribution of an estimator.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.