What does it mean for something to have good frequentist properties?

Question

I've often heard this phrase, but have never entirely understood what it means. The phrase "good frequentist properties" has ~2750 hits on google at present, 536 on scholar.google.com, and 4 on stats.stackexchange.com.

The closest thing I found to a clear definition comes from the final slide in this Stanford University presentation, which states

[T]he meaning of reporting 95% confidence intervals is that you “trap” the true parameter in 95% of the claims that you make, even across different estimation problems. This is the defining characteristic of estimation procedures with good frequentist properties: they hold up to scrutiny when repeatedly used.

Reflecting a bit on this, I assume that the phrase "good frequentist properties" implies some assessment of a Bayesian method, and in particular a Bayesian method of interval construction. I understand that Bayesian intervals are meant to contain the true value of the parameter with probability $p$. Frequentist intervals are meant to be constructed such that such that if the if the process of interval construction was repeated many times about $p*100\%$ of the intervals would contain the true value of the parameter. Bayesian intervals don't in general make any promises about what % of the intervals will cover the true value of the parameter. However, some Bayesian methods also happen to have the property that if repeated many times they cover the true value about $p*100\%$ of the time. When they have that property, we say they have "good frequentist properties".

Is that right? I figure that there must be more to it than that, since the phrase refers to good frequentist properties, rather than having a good frequentist property.

Answer 1

A tricky thing about good frequentist properties is that they are properties of a procedure rather than properties of a particular result or inference. A good frequentist procedure yields correct inferences on the specified proportion of cases in the long run, but a good Bayesian procedure is often the one that yields correct inferences in the individual case in question.

For example, consider a Bayesian procedure that is "good" in a general sense because it supplies a posterior probability distribution or credible interval that correctly represents combination of the evidence (likelihood function) with the prior probability distribution. If the prior contains accurate information (say, rather than empty opinion or some form of uninformative prior), that posterior or interval might result in better inference than a frequentist result from the same data. Better in the sense of leading to more accurate inference about this particular case or a narrower estimation interval because the procedure utilises a customised prior containing accurate information. In the long run the coverage percentage of the intervals and the correctness of inferences is influenced by the quality of each prior. Such a procedure will not have "good frequentist properties" because it is dependent on the quality of the prior and the prior is not appropriately customised in the long run accounting.

Notice that the procedure does not specify how the prior is to be obtained and so the long run accounting of performance would, presumably, assume any-old prior rather than a custom-designed prior for each case.

A Bayesian procedure can have good frequentist properties. For example, in many cases a Bayesian procedure with a recipe-provided uninformative prior will have fairly good to excellent frequentist properties. Those good properties would be an accident rather than design feature, and would be a straightforward consequence of such a procedure yielding similar intervals to the frequentist procedures.

Thus a Bayesian procedure can have superior inferential properties in an individual experiment while having poor frequentist properties in the long run. Equivalently, frequentist procedures with good long run frequentist properties often have poor performance in the case of individual experiments.

Answer 2

I would answer that your analysis is correct. To provide a few more insights, I would mention matching priors.

Matching priors are typically priors designed to build Bayesian models with a frequentist property. In particular, they are defined so that the obtained hpd intervals satisfy the frequentist coverage of confidence interval (so 95% of the 95% hpd contain the true values on the long run). Notice that, in 1d, there are analytical solutions: the Jeffreys priors are matching priors. In higher dimension, this is not necessary the case (to my knownledge, there is no result proving that this is never the case).

In practice, this matching principle is sometimes also applied to tune the value of some parameters of a model: ground truth data are used to optimize these parameters in the sense that their values maximise the frequentist coverage of the resulting credible intervals for the parameter of interest. From my own experimence, this may be a very subtle task.

Answer 3

If there is any contribution that I can make, let me add a clarification first, and then answer your question directly. There are a lot of confusion about the topic (frequestist properties of bayesian procedure), and even disagreement among specialists. The first misconception is "Bayesian intervals are meant to contain the true value of the parameter with probability $p$." If you are pure Bayesian (one that does not adopt any frequentist notion to evaluate the Bayesian procedure), there is no such thing as "true parameter". The main quantities of interested that are fixed parameters in the frequentist world are random variables in the Bayesian world . As a Bayesian, you do not recover the true value of the parameters, but the distribution of the "parameters", or their moments.

Now, to answer your question: no, it does not imply any assessement of the Bayesian method. Skiping the nuances and focusing in estimation procedure to keep it simple: the frequentism in statistics is that idea of estimating an unknown fixed quantity, or testing a hypothesis, and evaluating such procedure against a hypothetical repetition of it. You can adopt many criteria to evaluate a procedure. What makes it a frequentist criterium is that one cares about what happen if one adopts the same procedure over and over again. If you do so, you care about the frequentist properties. In other words: "what are the frequentist properties?" means "what happens if we repeat the procedure over and over?" Now, what makes such frequentist properties good is another layer of criteria. The most common frequentist properties that are considered good properties are consistency (in an estimation, if you keep sampling the estimator will converge to the fixed value you are estimating), efficiency (if you keep sampling, the variance of the estimator will go to zero, so you will be more and more accurate), coverage probability (in many repetitions of the procedure, a 95% confidence interval will contain the true value 95% of the time). The first two are called large sample properties, the third is the Neyman-genuinely frequentist property in the sense that it does not need to use asymptotic results necessarily. So, in sum, in the frequentist framework, there is a true and unknown value. You estimate it and you are always (except in a rare lucky accident) wrong in the estimation, but you are trying to save yourself by requiring that at least under a hypothetically indefinitely repetition of your estimation, you would be less and less wrong or you know you would be right a certain amount of times. I won't discuss if it makes sense or not, or the additional assumptions required to justify it, given it was not your questions. Conceptually, that is what frequentist properties refer to, and which good means in general in such context.

I will close by pointing you this paper, so that you judge by yourself if it makes sense and what it means a Bayesian procedure to have good frequentist properties (you will find more references there):

• Little, R., & others, (2011). Calibrated bayes, for statistics in general, and missing data in particular. Statistical Science, 26(2),162–174.