2
$\begingroup$

After reading Larry Wassermans blog on the difference between Bayesian and Frequentist inference I started to appreciate that frequency guarantees can be desirable regardless of the inference method that you choose, since some Bayesian inference methods have frequency guarantees.

With that said, assume that I am developing a prediction model for a Business application that will perform a large number of predictions over the next year. Furthermore, the Business requires a guarantee of the performance of this prediction model in order to manage risk.

Is a guarantee of the performance of a (Bayesian or Frequentist) prediction model a frequency guarantee of that prediction model? If all guarantees of performance are frequency guarantees should frequency guarantees not always be desirable for both Bayesians and Frequentists in this Business context?

As an example: if I am learning from a class of (classification) prediction models with finite VC dimension and more data than the corresponding sample complexity is available I have guarantees on the performance of the prediction model. However, these results from statistical learning theory are derived from the risk which involves an integral over the data which is fundamentally frequentist, i.e. a frequency guarantee. My intuition is that all guarantees about performance (future data) will involve an integral over the data and therefore Bayesians require frequency guarantees to guarantee performance, but I might be mistaken?

$\endgroup$
3
$\begingroup$

Is an estimate of the performance of a (Bayesian or Frequentist) prediction model a frequency guarantee of that prediction model?

No. Performance estimates do not guarantee performance; they estimate it. Also, most performance guarantees are asymptotic. Do you have an infinite amount of data?

Unbiased estimators are guaranteed to be perfectly accurate. They may perform poorly when measured, instead, by MSE. Models that perform well under MSE may be inaccurate.

Tools should be used based on business goals, not arbitrary measures such as MSE. Why not mean absolute deviation instead? A great estimator under the mean absolute deviation could be a terrible estimator under MSE. What does the business lose if you estimate too high or too low? What does that function look like?

A guarantee of coverage is a valuable insurance policy. To see why, consider how much damage repeatedly stopping an assembly line for false positives could do to productivity, morale, and output.

In other areas, it would be damaging due to the loss in precision needed to pay for the guarantee. It is an insurance policy. Do you need insurance, and are you willing to pay for it?

Frequentist methods tend to come with guarantees, such as "unbiasedness," or "coverage," or "bounding the percentage of false positives."

Their performance is guaranteed, usually as some process is repeated an infinite amount of time. There is no guarantee for specific performance on your specific data.

Guarantees only happen when you force guarantees to happen. When you do not, they do not happen, at least sometimes.

What I take from the Wasserman post is that Frequentist procedures usually do exceedingly well under Frequentist testing criteria and Bayesian ones do so under Bayesian testing criteria. They tend to do poorly, when tested against different criteria.

That is only logical. Both Bayesian and Frequentist methods have strong optimality properties. However, they are optimal with respect to different types of functions (generalized Bayes rules being the exception for both groups, when one exists).

If I perform an optimization using function $\mathcal{F}$, but then test the results under a very different function $\mathcal{B}$, it would be startling to find out that the estimators performed consistently well, excluding certain simple cases.

Bayesian predictions should not be treated as having guarantees.

To give a concrete example, consider a prediction based on a regression where all the standard requirements of ordinary least squares hold. The least-squares estimator will be BLUE, guaranteed. However, if there is prior information as to the location of the parameter, it will not be an admissible estimator. However, that is unfair to the Frequentist method in the sense that it is being judged against a tool with more information in it. It wasn't promising admissibility, it was promising it would be the MVUE.

Conversely, the Bayesian estimator might be the admissible estimator but would fail if bias is the standard of judgment. Think about all the outcry over biased estimators and racist outcomes from them.

No, estimates are not guarantees, even though in some cases, the results may act as a near de facto guarantee. Of course, a near guarantee is not a guarantee.

EDIT

In response to the comment, to get performance guarantees, you need to use a Frequentist method. They hold them by design. Bayesian predictive distributions and intervals minimize the K-L divergence from nature. The most commonly used Frequentist predictive interval minimizes the average K-L divergence. In so doing, it gets guaranteed performance levels but produces a result that may be far from nature with a given sample.

One of the more common errors regarding confidence intervals is speaking of them as credible intervals. However, under some priors and likelihoods, the boundaries are congruent. When that happens, although it is still an error, it is coincidentally true. The converse is also true in that case, the Bayesian interval inherits the Frequentist coverage probabilities.

Bayesian methods cannot guarantee coverage because they work in the parameter space, while Frequentist methods work in the sample space.

If you have a business goal, that goal should determine the success criteria and the tool used.

Although this example shows confidence and credible intervals, there is a small leap from those intervals to predictive ones. When you look at it, you will be able to see why you cannot mix and match the tools.

It is a simple problem with radically different intervals and the properties of those intervals are quite apart.

The frequentist intervals provide 0% and 41% credible sets in two cases, even though they are 70% confidence intervals, while the Bayesian 70% interval only provides 20% coverage under one value of the parameter.

The only cases where Bayesian intervals have incidentally equal coverage properties mostly happen in cases where doing that result on purpose would be undesirable on the Bayesian side.

$\endgroup$
1
  • $\begingroup$ I think I should have said "performance guarantees" rather than "performance estimates", because I am really wondering if in principle performance guarantees are possible without using frequency methods. Is any (performance) guarantee about future data necessarily going to be a frequentist guarantee? $\endgroup$ – mtedwards Aug 11 '20 at 7:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.