# What does it mean for something to have good frequentist properties?

I've often heard this phrase, but have never entirely understood what it means. The phrase "good frequentist properties" has ~2750 hits on google at present, 536 on scholar.google.com, and 4 on stats.stackexchange.com.

The closest thing I found to a clear definition comes from the final slide in this Stanford University presentation, which states

[T]he meaning of reporting 95% confidence intervals is that you “trap” the true parameter in 95% of the claims that you make, even across different estimation problems. This is the defining characteristic of estimation procedures with good frequentist properties: they hold up to scrutiny when repeatedly used.

Reflecting a bit on this, I assume that the phrase "good frequentist properties" implies some assessment of a Bayesian method, and in particular a Bayesian method of interval construction. I understand that Bayesian intervals are meant to contain the true value of the parameter with probability $p$. Frequentist intervals are meant to be constructed such that such that if the if the process of interval construction was repeated many times about $p*100\%$ of the intervals would contain the true value of the parameter. Bayesian intervals don't in general make any promises about what % of the intervals will cover the true value of the parameter. However, some Bayesian methods also happen to have the property that if repeated many times they cover the true value about $p*100\%$ of the time. When they have that property, we say they have "good frequentist properties".

Is that right? I figure that there must be more to it than that, since the phrase refers to good frequentist properties, rather than having a good frequentist property.

• I really like the way you thought out this question. In the early days Sir Harold Jeffreys tried to construct Bayesian posterior distributions that behaved like likelihood functions and hence had good frequentist properties. So it amounts to constructing a "uniform" prior distribution. The idea is that using such a prior means the prior is neutral and does not influence the inference. So this applies to more than just making credible intervals look like confidence intervals. But Jeffreys ran into some trouble because there were cases where the "uniform" prior was not proper. – Michael R. Chernick Dec 7 '16 at 7:22
• Improper means that the prior density does not integrate to 1. It seems that Jeffreys believed that the Bayesian method needed to be justified by agreeing with the frequentist method. Bayesians eventually rejected this notion because the value of the approach they claim is that there is prior information that influences the inference and so they prefer to use proper "informative" priors. – Michael R. Chernick Dec 7 '16 at 7:29
• @MichaelChernick: can you provide a precise reference about Jeffreys seeking frequentist properties for Bayes estimators? I have never heard of this story. And I also doubt Jeffreys was at all worried about using improper priors, they are all over Theory of Probability. – Xi'an Dec 7 '16 at 9:25
• I LOVE this question! – Alexis Dec 7 '16 at 22:55
• @Xi'an as a matter of fact, for the Beta-Binomial model it's the Haldane prior (which is improper) the one which leads to the frequentist estimate, not the Jeffreys prior (which is proper, in this case). I also have never heard that Jeffreys was looking for good frequentist properties: I thought he was looking for objective priors, and by objective he meant invariant under reparametrization. – DeltaIV Dec 8 '16 at 7:50

A tricky thing about good frequentist properties is that they are properties of a procedure rather than properties of a particular result or inference. A good frequentist procedure yields correct inferences on the specified proportion of cases in the long run, but a good Bayesian procedure is often the one that yields correct inferences in the individual case in question.

For example, consider a Bayesian procedure that is "good" in a general sense because it supplies a posterior probability distribution or credible interval that correctly represents combination of the evidence (likelihood function) with the prior probability distribution. If the prior contains accurate information (say, rather than empty opinion or some form of uninformative prior), that posterior or interval might result in better inference than a frequentist result from the same data. Better in the sense of leading to more accurate inference about this particular case or a narrower estimation interval because the procedure utilises a customised prior containing accurate information. In the long run the coverage percentage of the intervals and the correctness of inferences is influenced by the quality of each prior. Such a procedure will not have "good frequentist properties" because it is dependent on the quality of the prior and the prior is not appropriately customised in the long run accounting.

Notice that the procedure does not specify how the prior is to be obtained and so the long run accounting of performance would, presumably, assume any-old prior rather than a custom-designed prior for each case.

A Bayesian procedure can have good frequentist properties. For example, in many cases a Bayesian procedure with a recipe-provided uninformative prior will have fairly good to excellent frequentist properties. Those good properties would be an accident rather than design feature, and would be a straightforward consequence of such a procedure yielding similar intervals to the frequentist procedures.

Thus a Bayesian procedure can have superior inferential properties in an individual experiment while having poor frequentist properties in the long run. Equivalently, frequentist procedures with good long run frequentist properties often have poor performance in the case of individual experiments.

• I don't follow. Except for Empirical Bayes, in all Bayesian procedures I saw the prior is chosen independently of data. Thus, when applying such a procedure to multiple data sets coming from the same data generating process (this is the frequentist framework) the Bayesian will use the same likelihood function ( the data generating process is the same) and the same prior (the prior is independent of data in most Bayes procedures). Of course since the data change each time, the value of the likelihood changes, but its form is the same. Now, if each individual [1/2] – DeltaIV Dec 7 '16 at 7:43
• [2/2] estimate is more accurate, how can the whole procedure be less accurate? This is only possible if the Bayesian estimate is not always more accurate. However, since the prior is not customised to observed data, I'm not sure what makes it more or less accurate for each single case and/or "on average". – DeltaIV Dec 7 '16 at 7:49
• @DeltaV I think that you are dealing with the wrong reference set. Frequentist properties of a procedure relate to long run performance of the procedure applied in all new cases, not just to repeats of the particular experiment. That's why confidence interval procedures for binomial proportions have to work for all values of the parameter, not just for the value relevant to any particular instance where the procedure is used. That type of 'long run' means that the customised prior that is appropriate to the case in question will be inappropriate for the long run. – Michael Lew Dec 7 '16 at 20:07
• you are right that a frequentist confidence procedure must have the nominal coverage for all values of the unknown parameter. This was clearly specified by Newman & Pearson, and it's often overlooked today. However, when you choose the prior, you don't know which is the "true" value of the parameter. You only have your sample, and the prior should be independent of the sample. Thus I still don't see clearly how you could customize the prior based on the sample. Can you make a practical example? – DeltaIV Dec 7 '16 at 22:12
• @DeltaIV If I know that the current parameter of interest has been estimated in the previous study then I can shape an informative prior based on that estimate. That prior will be appropriate for this current analysis, but there is not an equivalent appropriate informative prior available for the notional set of applications of the method in the long run. Thus the analysis can have much better properties in the isolated real case than it would appear to have in the frequentist long run. – Michael Lew Dec 8 '16 at 6:43

I would answer that your analysis is correct. To provide a few more insights, I would mention matching priors.

Matching priors are typically priors designed to build Bayesian models with a frequentist property. In particular, they are defined so that the obtained hpd intervals satisfy the frequentist coverage of confidence interval (so 95% of the 95% hpd contain the true values on the long run). Notice that, in 1d, there are analytical solutions: the Jeffreys priors are matching priors. In higher dimension, this is not necessary the case (to my knownledge, there is no result proving that this is never the case).

In practice, this matching principle is sometimes also applied to tune the value of some parameters of a model: ground truth data are used to optimize these parameters in the sense that their values maximise the frequentist coverage of the resulting credible intervals for the parameter of interest. From my own experimence, this may be a very subtle task.

If there is any contribution that I can make, let me add a clarification first, and then answer your question directly. There are a lot of confusion about the topic (frequestist properties of bayesian procedure), and even disagreement among specialists. The first misconception is "Bayesian intervals are meant to contain the true value of the parameter with probability $p$." If you are pure Bayesian (one that does not adopt any frequentist notion to evaluate the Bayesian procedure), there is no such thing as "true parameter". The main quantities of interested that are fixed parameters in the frequentist world are random variables in the Bayesian world . As a Bayesian, you do not recover the true value of the parameters, but the distribution of the "parameters", or their moments.

Now, to answer your question: no, it does not imply any assessement of the Bayesian method. Skiping the nuances and focusing in estimation procedure to keep it simple: the frequentism in statistics is that idea of estimating an unknown fixed quantity, or testing a hypothesis, and evaluating such procedure against a hypothetical repetition of it. You can adopt many criteria to evaluate a procedure. What makes it a frequentist criterium is that one cares about what happen if one adopts the same procedure over and over again. If you do so, you care about the frequentist properties. In other words: "what are the frequentist properties?" means "what happens if we repeat the procedure over and over?" Now, what makes such frequentist properties good is another layer of criteria. The most common frequentist properties that are considered good properties are consistency (in an estimation, if you keep sampling the estimator will converge to the fixed value you are estimating), efficiency (if you keep sampling, the variance of the estimator will go to zero, so you will be more and more accurate), coverage probability (in many repetitions of the procedure, a 95% confidence interval will contain the true value 95% of the time). The first two are called large sample properties, the third is the Neyman-genuinely frequentist property in the sense that it does not need to use asymptotic results necessarily. So, in sum, in the frequentist framework, there is a true and unknown value. You estimate it and you are always (except in a rare lucky accident) wrong in the estimation, but you are trying to save yourself by requiring that at least under a hypothetically indefinitely repetition of your estimation, you would be less and less wrong or you know you would be right a certain amount of times. I won't discuss if it makes sense or not, or the additional assumptions required to justify it, given it was not your questions. Conceptually, that is what frequentist properties refer to, and which good means in general in such context.

I will close by pointing you this paper, so that you judge by yourself if it makes sense and what it means a Bayesian procedure to have good frequentist properties (you will find more references there):

• Little, R., & others, (2011). Calibrated bayes, for statistics in general, and missing data in particular. Statistical Science, 26(2),162–174.