20
$\begingroup$

The Gauss-Markov theorem tells us that the OLS estimator is the best linear unbiased estimator for the linear regression model.

But suppose I don't care about linearity and unbiasedness. Then is there some other (possible nonlinear/biased) estimator for the linear regression model which is the most efficient under the Gauss-Markov assumptions or some other general set of assumptions?

There is of course one standard result: OLS itself is the best unbiased estimator if in addition to the Gauss-Markov assumptions we also assume that the errors are normally distributed. For some other particular distribution of errors I could compute the corresponding maximum-likelihood estimator.

But I was wondering if there is some estimator which is better-than-OLS in some relatively general set of circumstances?

$\endgroup$

4 Answers 4

20
$\begingroup$

Unbiased estimates are typical in introductory statistics courses because they are: 1) classic, 2) easy to analyze mathematically. The Cramer-Rao lower bound is one of the main tools for 2). Away from unbiased estimates there is possible improvement. The bias-variance trade off is an important concept in statistics for understanding how biased estimates can be better than unbiased estimates.

Unfortunately, biased estimators are typically harder to analyze. In regression, much of the research in the past 40 years has been about biased estimation. This began with ridge regression (Hoerl and Kennard, 1970). See Frank and Friedman (1996) and Burr and Fry (2005) for some review and insights.

The bias-variance tradeoff becomes more important in high-dimensions, where the number of variables is large. Charles Stein surprised everyone when he proved that in the Normal means problem the sample mean is no longer admissible if $p \geq 3$ (see Stein, 1956). The James-Stein estimator (James and Stein 1961) was the first example of an estimator that dominates the sample mean. However, it is also inadmissible.

An important part of the bias-variance problem is determining how bias should be traded off. There is no single “best” estimator. Sparsity has been an important part of research in the past decade. See Hesterberg et al. (2008) for a partial review.

Most of the estimators referenced above are non-linear in $Y$. Even ridge regression is non-linear once the data is used to determine the ridge parameter.

$\endgroup$
3
  • 1
    $\begingroup$ @chl seconded. Great overview. $\endgroup$
    – mpiktas
    Mar 1, 2011 at 11:34
  • 2
    $\begingroup$ One of my favourite admissible estimators: a single arbitrarily chosen point of the parameter space which is not an impossible value :) $\endgroup$ May 20, 2011 at 14:03
  • $\begingroup$ @probabilityislogic Those often are admissible, but what you wrote is not 100% true if we (somewhat) artificially restrict the parameter space (e.g. we only allow for heads-heads or tails-tails coins). That a possible constant is not always admissible surprised me. $\endgroup$
    – Dave
    Jun 29, 2021 at 21:30
9
$\begingroup$

I don't know if you are OK with the Bayes Estimate? If yes, then depending on the Loss function you can obtain different Bayes Estimates. A theorem by Blackwell states that Bayes Estimates are never unbiased. A decision theoretic argument states that every admissible rule ((i.e. or every other rule against which it is compared, there is a value of the parameter for which the the risk of the present rule is (strictly) less than that of rule against which it's being compared)) is a (generalized) Bayes rule.

James-Stein Estimators are another class of estimators (which can be derived by Bayesian methods asymptotically) which are better than OLS in many cases.

OLS can be inadmissible in many situations and James-Stein Estimator is an example. (also called Stein's paradox).

$\endgroup$
24
  • $\begingroup$ Thanks for the pointers. Will need to hit the library to make sense of it all. $\endgroup$ Sep 23, 2010 at 14:53
  • 2
    $\begingroup$ @suncoolsu, that's not the typical definition of admissibility. The one you've given is (much) stronger. An admissible estimator is one that is not uniformly dominated, i.e., for every other rule against which it is compared, there is a value of the parameter for which the the risk of the present rule is (strictly) less than that of rule against which it's being compared. Conversely, an inadmissible estimator is one that is (weakly) dominated by some other estimator for every value of the parameter and is strictly dominated for at least one value by that same estimator. $\endgroup$
    – cardinal
    Feb 27, 2011 at 15:24
  • $\begingroup$ @cardinal Yup. You are right. I will correct it. $\endgroup$
    – suncoolsu
    Feb 28, 2011 at 0:11
  • $\begingroup$ @cardinal. Using math is much easier than simplifying it in plain English. But that is just me. Thanks for the correction @cardinal $\endgroup$
    – suncoolsu
    Feb 28, 2011 at 0:17
  • 1
    $\begingroup$ @suncoolsu asymptotic unbiasedness is very different from the usual sense of "unbiased". Any reasonable estimate should be asymptotically unbiased. One more note: shouldn't the statement about admissible estimators be the other way around? ie every admissible estimator is generalized Bayes. $\endgroup$
    – vqv
    Feb 28, 2011 at 4:15
5
$\begingroup$

There is a nice review paper by Kay and Eldar on biased estimation for the purpose of finding estimators with minimum mean square error.

$\endgroup$
0
0
$\begingroup$

“Best” in BLUE means the minimum variance.

Variance is a non-negative quantity, so its lowest value is zero.

If you estimate the coefficients by picking a constant every time, then your estimator has zero variance.

In other words, just estimating the coefficients as zero every time, regardless of the data, is a zero-variance estimator.

Further, this estimator is a linear combination of elements of $y$, so this is a linear estimator.

Somewhat related, it came to my great surprise that this approach need not result in an estimator that is “admissible” under square loss.

While this is technically a way of doing estimation, it is a silly approach by human standards. For this reason, I believe that it is important to put some conditions of what you want out of an estimator if you don’t just want the minimum-variance estimator. For instance, perhaps you’re willing to drop linearity and incur some bias, but then you demand that the estimator at least be consistent. Perhaps you do not demand consistency, but you want the minimum MSE. Perhaps you want the estimator that gives the lowest maximum risk across all possible parameter values (a so-called “minimax” estimator).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.