12
$\begingroup$

The Gauss-Markov theorem tells us that the OLS estimator is the best linear unbiased estimator for the linear regression model.

But suppose I don't care about linearity and unbiasedness. Then is there some other (possible nonlinear/biased) estimator for the linear regression model which is the most efficient under the Gauss-Markov assumptions or some other general set of assumptions?

There is of course one standard result: OLS itself is the best unbiased estimator if in addition to the Gauss-Markov assumptions we also assume that the errors are normally distributed. For some other particular distribution of errors I could compute the corresponding maximum-likelihood estimator.

But I was wondering if there is some estimator which is better-than-OLS in some relatively general set of circumstances?

$\endgroup$
16
$\begingroup$

Unbiased estimates are typical in introductory statistics courses because they are: 1) classic, 2) easy to analyze mathematically. The Cramer-Rao lower bound is one of the main tools for 2). Away from unbiased estimates there is possible improvement. The bias-variance trade off is an important concept in statistics for understanding how biased estimates can be better than unbiased estimates.

Unfortunately, biased estimators are typically harder to analyze. In regression, much of the research in the past 40 years has been about biased estimation. This began with ridge regression (Hoerl and Kennard, 1970). See Frank and Friedman (1996) and Burr and Fry (2005) for some review and insights.

The bias-variance tradeoff becomes more important in high-dimensions, where the number of variables is large. Charles Stein surprised everyone when he proved that in the Normal means problem the sample mean is no longer admissible if $p \geq 3$ (see Stein, 1956). The James-Stein estimator (James and Stein 1961) was the first example of an estimator that dominates the sample mean. However, it is also inadmissible.

An important part of the bias-variance problem is determining how bias should be traded off. There is no single “best” estimator. Sparsity has been an important part of research in the past decade. See Hesterberg et al. (2008) for a partial review.

Most of the estimators referenced above are non-linear in $Y$. Even ridge regression is non-linear once the data is used to determine the ridge parameter.

$\endgroup$
  • 1
    $\begingroup$ @chl seconded. Great overview. $\endgroup$ – mpiktas Mar 1 '11 at 11:34
  • 1
    $\begingroup$ One of my favourite admissible estimators: a single arbitrarily chosen point of the parameter space which is not an impossible value :) $\endgroup$ – probabilityislogic May 20 '11 at 14:03
8
$\begingroup$

I don't know if you are OK with the Bayes Estimate? If yes, then depending on the Loss function you can obtain different Bayes Estimates. A theorem by Blackwell states that Bayes Estimates are never unbiased. A decision theoretic argument states that every admissible rule ((i.e. or every other rule against which it is compared, there is a value of the parameter for which the the risk of the present rule is (strictly) less than that of rule against which it's being compared)) is a (generalized) Bayes rule.

James-Stein Estimators are another class of estimators (which can be derived by Bayesian methods asymptotically) which are better than OLS in many cases.

OLS can be inadmissible in many situations and James-Stein Estimator is an example. (also called Stein's paradox).

$\endgroup$
  • $\begingroup$ Thanks for the pointers. Will need to hit the library to make sense of it all. $\endgroup$ – Jyotirmoy Bhattacharya Sep 23 '10 at 14:53
  • 1
    $\begingroup$ @suncoolsu, that's not the typical definition of admissibility. The one you've given is (much) stronger. An admissible estimator is one that is not uniformly dominated, i.e., for every other rule against which it is compared, there is a value of the parameter for which the the risk of the present rule is (strictly) less than that of rule against which it's being compared. Conversely, an inadmissible estimator is one that is (weakly) dominated by some other estimator for every value of the parameter and is strictly dominated for at least one value by that same estimator. $\endgroup$ – cardinal Feb 27 '11 at 15:24
  • $\begingroup$ @cardinal Yup. You are right. I will correct it. $\endgroup$ – suncoolsu Feb 28 '11 at 0:11
  • $\begingroup$ @cardinal. Using math is much easier than simplifying it in plain English. But that is just me. Thanks for the correction @cardinal $\endgroup$ – suncoolsu Feb 28 '11 at 0:17
  • 1
    $\begingroup$ @suncoolsu asymptotic unbiasedness is very different from the usual sense of "unbiased". Any reasonable estimate should be asymptotically unbiased. One more note: shouldn't the statement about admissible estimators be the other way around? ie every admissible estimator is generalized Bayes. $\endgroup$ – vqv Feb 28 '11 at 4:15
4
$\begingroup$

There is a nice review paper by Kay and Eldar on biased estimation for the purpose of finding estimators with minimum mean square error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.