The concept of efficiency

Question

I have some problems in understanding the concept of efficiency as related to an estimator. My sources (Mukhopadhyay, 2000 and Casella, Berger, 2002) do not treat this argument as I expected since they analyse only the concept of asymptotic efficiency.

I do not understand if there exist a concept of efficiency valid also (or only) in finite samples. Or if efficiency is not a per se concept and it is used only to compare estimators, talking about more efficient estimator and so on.

What I know is that an estimator is efficient if it reaches the Cramér-Rao Lower Bound. But this is a characterization, it is not a definition. And, moreover, Cramér-Rao inequality refers only to a subset of estimators, those which are unbiased for a certain $\tau(\theta)$. The concept of efficiency is meaningful only in the case of unbiased estimators?

If someone could provide me some sources or a brief excursus about the concept of efficiency and efficient estimators, I will be grateful.

Answer 1

Efficiency is a "per se" concept in the sense that it is a measure of how variable (and biased) the estimator is from the "true" parameter. There is an actual numeric value for efficiency associated with a given estimator at a given sample-size for a given loss function. This actual number is related to the estimator AND the sample-size AND the loss function.

Asymptotic efficiency looks at how efficient the estimator is as the sample size increases. More important is how rapidly the estimator becomes efficient but this can be more difficult to determine.

Relative efficiency looks at how efficient the estimator is relative to an alternative estimator (typically at a GIVEN sample-size).

Efficiency requires the specification of some loss function. Originally, this was variance when only unbiased estimators were considered. These days, this is most often MSE (mean-squared-error which accounts for bias and variability). Other loss-functions can be used. The classical Cramer-Rao bound was for unbiased estimators only but was extended to many of these other loss functions (most especially for MSE loss).

An important adjunct concept is admissibility and domination of estimators.

The Wikipedia entry has many links.

Answer 2

I wonder at the global relevance of a concept of efficiency outside [and even inside] the restricted case of unbiased estimators. The general (frequentist) version is that the variance of an estimator $δ$ of [any transform of] $θ$ with bias b(θ) is $$ I(θ)⁻¹ (1+b'(θ))² $$ while a Bayesian version is the van Trees inequality on the integrated squared error loss $$ (\mathbb{E}(I(θ))+I(π))⁻¹ $$ where $I(θ)$ and $I(π)$ are the Fisher information and the prior entropy, respectively. But this opens a whole can of worms, in my opinion since

1. establishing that a given estimator is efficient requires computing both the bias and the variance of that estimator, not an easy task when considering a Bayes estimator or even the James-Stein estimator. I actually do not know if any of the estimators dominating the standard Normal mean estimator has been shown to be efficient (although there exist results for closed form expressions of the James-Stein estimator quadratic risk, including one of mine in the Canadian Journal of Statistics). Or is there a result indicating that a (any?) proper Bayes estimator associated with the quadratic loss is by default efficient in either the first or second sense?
2. while the initial Fréchet-Darmois-Cramèr-Rao bound is restricted to unbiased estimators (i.e., $b(θ)≡0$) and unable to produce efficient estimators in all settings but for the natural parameter in the setting of exponential families, moving to the general case means there exists one efficiency notion for every bias function $b(θ)$, which makes the notion quite weak, while not necessarily producing efficient estimators anyway, the major impediment to taking this notion seriously;
3. moving from the variance to the squared error loss is not more "natural" than using any [other] convex combination of variance and squared bias, creating a whole new class of optimalities;
4. I never got into the van Trees inequality so cannot say much, except that the comparison between various priors is delicate since the integrated risks are against different parameter measures.

Answer 3

Yes an efficient estimator is one that attains the CRB, thus only unbiased estimators are considered. It's not a characterization it's a definition.