How do I know which method of parameter estimation to choose?

Question

There are quite a few methods for parameter estimation out there. MLE, UMVUE, MoM, decision-theoretic, and others all seem like they have a fairly logical case for why they are useful for parameter estimation. Is any one method better than the others, or is it just a matter of how we define what the "best fitting" estimator is (similar to how minimizing orthogonal errors produces different estimates from an ordinary least squares approach)?

Answer 1

There's a slight confusion of two things here: methods for deriving estimators, and criteria for evaluating estimators. Maximum likelihood (ML) and method-of-moments (MoM) are ways of deriving estimators; Uniformly minimum variance unbiasedness (UMVU) and decision theory are criteria for evaluating different estimators once you have them, but they won't tell you how to derive them.

Of the methods for deriving estimators, ML usually produces estimators that are more efficient (i.e. lower variance) than MoM if you know the model under which your data were derived (the 'data-generating process' (DGP) in the jargon). But MoM makes fewer assumptions about the model; as its name implies, it only uses one or more moments, usually just the mean or just the mean and variance, so it's sometimes more robust if you're not sure about the DGP. There can be more than one MoM estimator for the same problem, while if you know the DGP, there is only one ML estimator.

Of the methods for evaluating estimators, decision-theoretic depends on having a loss function to use to judge your estimator, although the results can be fairly robust to a range of 'reasonable' loss functions. UMVU estimators often don't even exist; in many cases there is no unbiased estimator that always has minimum variance. And the criterion of unbiasedness is also of questionable usefulness, as it's not invariant to transformations. For example, would you prefer an unbiased estimator of the odds ratio, or of the log odds ratio? The two will be different.

Answer 2

I would suggest that the type of estimator depends on a few things:

1. What are the consequences of getting the estimate wrong? (e.g. is it less bad if your estimator is too high, compared to being too low? or are you indifferent about the direction of error? if an error is twice as big, is this twice as bad? is it percentage error or absolute error that is important? Is the estimation only intermediate step that is required for prediction? is large sample behaviour more or less important than small sample behaviour?)
2. What is your prior information about the quantity you are estimating? (e.g. how is the data functionally related to your quantity? do you know if the quantity is positive? discrete? have you estimated this quantity before? how much data do you have? Is there any "group invariance" structure in your data?)
3. What software do you have? (e.g. no good suggesting MCMC if you don't have the software to do it, or using a GLMM if you don't know how to do it.)

The first two points are context specific, and by thinking about your specific application, you will generally be able to define certain properties that you would like your estimator to have. You then choose the estimator which you can actually calculate, which has as many of the properties which you want it to have.

I think the lack of context that a teaching course has with estimation, means that often "default" criterion are used, similarly for prior information (the most obvious "default" being that you know the sampling distribution of your data). Having said that, some of the default methods are good, especially if you don't know enough about the context. But if you do know the context, and you have the tools to incorporate that context, then you should, for otherwise you may get counter-intuitive results (because of what you ignored).

The I'm not a big fan of MVUE as a general rule, because you often need to sacrifice too much variance to get unbiased-ness. For example, imagine you are throwing darts at a dartboard, and you want to hit to the bulls-eye. Supposing that the maximum deviation from the bulls-eye is 6cm for a particular throwing strategy, but the center of the dart points is 1cm above of the bullseye. This is not MVUE, because the center should be on the bullseye. But suppose that in order to shift the distribution down 1cm (on the average), you have to increase your radius to at least 10cm (so the maximum error is now 10cm, and not 6cm). This is the kind of thing that can happen with MVUE, unless the variance is already small. Suppose I was a much more accurate throw, and could narrow my error to 0.1cm. Now the bias really matters, because I will never hit the bullseye!

In short, for me, bias only matters when it is small compared to the variance. And you will usually only get small variances when you have a large sample.