The concept of efficiency

I have some problems in understanding the concept of efficiency as related to an estimator. My sources (Mukhopadhyay, 2000 and Casella, Berger, 2002) do not treat this argument as I expected since they analyse only the concept of asymptotic efficiency.

I do not understand if there exist a concept of efficiency valid also (or only) in finite samples. Or if efficiency is not a per se concept and it is used only to compare estimators, talking about more efficient estimator and so on.

What I know is that an estimator is efficient if it reaches the Cramér-Rao Lower Bound. But this is a characterization, it is not a definition. And, moreover, Cramér-Rao inequality refers only to a subset of estimators, those which are unbiased for a certain $\tau(\theta)$. The concept of efficiency is meaningful only in the case of unbiased estimators?

If someone could provide me some sources or a brief excursus about the concept of efficiency and efficient estimators, I will be grateful.

• Defining efficiency in terms of the Cramer-Rao lower bound is perfectly valid, and it makes sense for finite samples. You could extend this to unbiased estimators by talking instead about minimum mean squared error (or use some other loss function), but perhaps the math doesn't work as well. I'm guessing here but it may deal with the use of the Cauchy-Schwarz inequality. – dsaxton Jan 5 '16 at 18:57

Efficiency is a "per se" concept in the sense that it is a measure of how variable (and biased) the estimator is from the "true" parameter. There is an actual numeric value for efficiency associated with a given estimator at a given sample-size for a given loss function. This actual number is related to the estimator AND the sample-size AND the loss function.

Asymptotic efficiency looks at how efficient the estimator is as the sample size increases. More important is how rapidly the estimator becomes efficient but this can be more difficult to determine.

Relative efficiency looks at how efficient the estimator is relative to an alternative estimator (typically at a GIVEN sample-size).

Efficiency requires the specification of some loss function. Originally, this was variance when only unbiased estimators were considered. These days, this is most often MSE (mean-squared-error which accounts for bias and variability). Other loss-functions can be used. The classical Cramer-Rao bound was for unbiased estimators only but was extended to many of these other loss functions (most especially for MSE loss).

The Wikipedia entry has many links.

• This was a very well-written response to the question - thank you! – Matt Brems Jan 5 '16 at 22:31
• Thank you for this clarification. Thus, efficiency is an estimator feature as sufficiency or completeness? I mean, there are some properties related to efficient estimators, such as uniqueness, exponential families properties, ... ? – PhDing Jan 6 '16 at 8:47

I wonder at the global relevance of a concept of efficiency outside [and even inside] the restricted case of unbiased estimators. The general (frequentist) version is that the variance of an estimator $$δ$$ of [any transform of] $$θ$$ with bias b(θ) is $$I(θ)⁻¹ (1+b'(θ))²$$ while a Bayesian version is the van Trees inequality on the integrated squared error loss $$(\mathbb{E}(I(θ))+I(π))⁻¹$$ where $$I(θ)$$ and $$I(π)$$ are the Fisher information and the prior entropy, respectively. But this opens a whole can of worms, in my opinion since

1. establishing that a given estimator is efficient requires computing both the bias and the variance of that estimator, not an easy task when considering a Bayes estimator or even the James-Stein estimator. I actually do not know if any of the estimators dominating the standard Normal mean estimator has been shown to be efficient (although there exist results for closed form expressions of the James-Stein estimator quadratic risk, including one of mine in the Canadian Journal of Statistics). Or is there a result indicating that a (any?) proper Bayes estimator associated with the quadratic loss is by default efficient in either the first or second sense?
2. while the initial Fréchet-Darmois-Cramèr-Rao bound is restricted to unbiased estimators (i.e., $$b(θ)≡0$$) and unable to produce efficient estimators in all settings but for the natural parameter in the setting of exponential families, moving to the general case means there exists one efficiency notion for every bias function $$b(θ)$$, which makes the notion quite weak, while not necessarily producing efficient estimators anyway, the major impediment to taking this notion seriously;
3. moving from the variance to the squared error loss is not more "natural" than using any [other] convex combination of variance and squared bias, creating a whole new class of optimalities;
4. I never got into the van Trees inequality so cannot say much, except that the comparison between various priors is delicate since the integrated risks are against different parameter measures.

Yes an efficient estimator is one that attains the CRB, thus only unbiased estimators are considered. It's not a characterization it's a definition.

• If you read the answer above, it says that CRLB can be used also with other loss functions than the variance, i.e. MSE and others. Thus, I think the definition of efficiency can be attributed also to biased estimators – PhDing Jan 6 '16 at 11:46
• The definition of estimator's efficiency require the inspected estimator to be at least locally unbiased in a region of interest. If an MSE of an unbiased estimator equals the Cramer Rao bound then it is efficient. Otherwise, it's not. – Nir Regev Jan 6 '16 at 17:36
• Thus, to be efficient, being unbiased is a necessary condition? – PhDing Jan 6 '16 at 17:59
• Ok thanks. So the idea is that an efficient estimator must be unbiased but I can compute efficiency also for biased estimators? – PhDing Jan 6 '16 at 18:08
• No. You can't. You can compute it's MSE and compare it with the bound, which lacks a real mathematical meaning. Efficiency is a property of an estimator. Either it is efficient or not. – Nir Regev Jan 6 '16 at 18:10