Is an estimate of the performance of a (Bayesian or Frequentist) prediction model a frequency guarantee of that prediction model?
No. Performance estimates do not guarantee performance; they estimate it. Also, most performance guarantees are asymptotic. Do you have an infinite amount of data?
Unbiased estimators are guaranteed to be perfectly accurate. They may perform poorly when measured, instead, by MSE. Models that perform well under MSE may be inaccurate.
Tools should be used based on business goals, not arbitrary measures such as MSE. Why not mean absolute deviation instead? A great estimator under the mean absolute deviation could be a terrible estimator under MSE. What does the business lose if you estimate too high or too low? What does that function look like?
A guarantee of coverage is a valuable insurance policy. To see why, consider how much damage repeatedly stopping an assembly line for false positives could do to productivity, morale, and output.
In other areas, it would be damaging due to the loss in precision needed to pay for the guarantee. It is an insurance policy. Do you need insurance, and are you willing to pay for it?
Frequentist methods tend to come with guarantees, such as "unbiasedness," or "coverage," or "bounding the percentage of false positives."
Their performance is guaranteed, usually as some process is repeated an infinite amount of time. There is no guarantee for specific performance on your specific data.
Guarantees only happen when you force guarantees to happen. When you do not, they do not happen, at least sometimes.
What I take from the Wasserman post is that Frequentist procedures usually do exceedingly well under Frequentist testing criteria and Bayesian ones do so under Bayesian testing criteria. They tend to do poorly, when tested against different criteria.
That is only logical. Both Bayesian and Frequentist methods have strong optimality properties. However, they are optimal with respect to different types of functions (generalized Bayes rules being the exception for both groups, when one exists).
If I perform an optimization using function $\mathcal{F}$, but then test the results under a very different function $\mathcal{B}$, it would be startling to find out that the estimators performed consistently well, excluding certain simple cases.
Bayesian predictions should not be treated as having guarantees.
To give a concrete example, consider a prediction based on a regression where all the standard requirements of ordinary least squares hold. The least-squares estimator will be BLUE, guaranteed. However, if there is prior information as to the location of the parameter, it will not be an admissible estimator. However, that is unfair to the Frequentist method in the sense that it is being judged against a tool with more information in it. It wasn't promising admissibility, it was promising it would be the MVUE.
Conversely, the Bayesian estimator might be the admissible estimator but would fail if bias is the standard of judgment. Think about all the outcry over biased estimators and racist outcomes from them.
No, estimates are not guarantees, even though in some cases, the results may act as a near de facto guarantee. Of course, a near guarantee is not a guarantee.
EDIT
In response to the comment, to get performance guarantees, you need to use a Frequentist method. They hold them by design. Bayesian predictive distributions and intervals minimize the K-L divergence from nature. The most commonly used Frequentist predictive interval minimizes the average K-L divergence. In so doing, it gets guaranteed performance levels but produces a result that may be far from nature with a given sample.
One of the more common errors regarding confidence intervals is speaking of them as credible intervals. However, under some priors and likelihoods, the boundaries are congruent. When that happens, although it is still an error, it is coincidentally true. The converse is also true in that case, the Bayesian interval inherits the Frequentist coverage probabilities.
Bayesian methods cannot guarantee coverage because they work in the parameter space, while Frequentist methods work in the sample space.
If you have a business goal, that goal should determine the success criteria and the tool used.
Although this example shows confidence and credible intervals, there is a small leap from those intervals to predictive ones. When you look at it, you will be able to see why you cannot mix and match the tools.
It is a simple problem with radically different intervals and the properties of those intervals are quite apart.
The frequentist intervals provide 0% and 41% credible sets in two cases, even though they are 70% confidence intervals, while the Bayesian 70% interval only provides 20% coverage under one value of the parameter.
The only cases where Bayesian intervals have incidentally equal coverage properties mostly happen in cases where doing that result on purpose would be undesirable on the Bayesian side.
