Frequentist confidence intervals bound the rate of false positives (Type I errors), and guarantee their coverage will be bounded below by the confidence parameter, even in the worst case. Bayesian credibility intervals don't.

So if the thing you care about is false positives and you need to bound them, confidence intervals are the the approach that you'll want to use.

For example, let's say you have an evil king with a court of 100 courtiers and courtesans and he wants to play a cruel statistical game with them. The king has a bag of a trillion fair coins, plus one unfair coin whose heads probability is 10%. He's going to perform the following game. First, he'll draw a coin uniformly at random from the bag.

Then the coin will be passed around a room of 100 people and each one will be forced to do an experiment on it, privately, and then each person will state a 95% uncertainty interval on what they think the coin's heads probability is.

Anybody who gives an interval that represents a false positive -- i.e. an interval that doesn't cover the true value of the heads probability -- will be beheaded.

If we wanted to express the /a posteriori/ probability distribution function of the coin's weight, then of course a credibility interval is what does that. The answer will always be the interval [0.5, 0.5] irrespective of outcome. Even if you flip zero heads or one head, you'll still say [0.5, 0.5] because it's a heck of a lot more probable that the king drew a fair coin and you had a 1/1024 day getting ten heads in a row, than that the king drew the unfair coin.

So this is not a good idea for the courtiers and courtesans to use! Because when the unfair coin is drawn, the whole room (all 100 people) will be wrong and they'll all get beheaded.

In this world where the most important thing is false positives, what we need is an absolute guarantee that the rate of false positives will be less than 5%, no matter which coin is drawn. Then we need to use a confidence interval, like Blyth-Still-Casella or Clopper-Pearson, that works and provides at least 95% coverage irrespective of the true value of the parameter, even in the worst case. If everybody uses this method instead, then no matter which coin is drawn, at the end of the day we can guarantee that the expected number of wrong people will be no more than five.

So the point is: if your criterion requires bounding false positives (or equivalently, guaranteeing coverage), you gotta go with a confidence interval. That's what they do. Credibility intervals may be a more intuitive way of expressing uncertainty, they may perform pretty well from a frequentist analysis, but they are not going to provide the guaranteed bound on false positives you'll get when you go asking for it.

(Of course if you also care about false negatives, you'll need a method that makes guarantees about those too...)

Frequentist confidence intervals bound the rate of false positives (Type I errors), and guarantee their coverage will be bounded below by the confidence parameter, even in the worst case. Bayesian credibility intervals don't.

So if the thing you care about is false positives and you need to bound them, confidence intervals are the approach that you'll want to use.

For example, let's say you have an evil king with a court of 100 courtiers and courtesans and he wants to play a cruel statistical game with them. The king has a bag of a trillion fair coins, plus one unfair coin whose heads probability is 10%. He's going to perform the following game. First, he'll draw a coin uniformly at random from the bag.

Then the coin will be passed around a room of 100 people and each one will be forced to do an experiment on it, privately, and then each person will state a 95% uncertainty interval on what they think the coin's heads probability is.

Anybody who gives an interval that represents a false positive – i.e. an interval that doesn't cover the true value of the heads probability – will be beheaded.

If we wanted to express the a posteriori probability distribution function of the coin's weight, then of course a credibility interval is what does that. The answer will always be the interval [0.5, 0.5] irrespective of outcome. Even if you flip zero heads or one head, you'll still say [0.5, 0.5] because it's a heck of a lot more probable that the king drew a fair coin and you had a 1/1024 day getting ten heads in a row, than that the king drew the unfair coin.

So this is not a good idea for the courtiers and courtesans to use! Because when the unfair coin is drawn, the whole room (all 100 people) will be wrong and they'll all get beheaded.

In this world where the most important thing is false positives, what we need is an absolute guarantee that the rate of false positives will be less than 5%, no matter which coin is drawn. Then we need to use a confidence interval, like Blyth-Still-Casella or Clopper-Pearson, that works and provides at least 95% coverage irrespective of the true value of the parameter, even in the worst case. If everybody uses this method instead, then no matter which coin is drawn, at the end of the day we can guarantee that the expected number of wrong people will be no more than five.

So the point is: if your criterion requires bounding false positives (or equivalently, guaranteeing coverage), you gotta go with a confidence interval. That's what they do. Credibility intervals may be a more intuitive way of expressing uncertainty, they may perform pretty well from a frequentist analysis, but they are not going to provide the guaranteed bound on false positives you'll get when you go asking for it.

(Of course if you also care about false negatives, you'll need a method that makes guarantees about those too...)

Frequentist confidence intervals bound the rate of false positives (Type I errors), and guarantee their coverage will be bounded below by the confidence parameter, even in the worst case. Bayesian credibility intervals don't.

So if the thing you care about is false positives and you need to bound them, confidence intervals are the the approach that you'll want to use.

For example, let's say you have an evil king with a court of 100 courtiers and courtesans and he wants to play a cruel statistical game with them. The king has a bag of a trillion fair coins, plus one unfair coin whose heads probability is 10%. He's going to perform the following game. First, he'll draw a coin uniformly at random from the bag.

Then the coin will be passed around a room of 100 people and each one will be forced to do an experiment on it, privately, and then each person will state a 95% uncertainty interval on what they think the coin's heads probability is.

Anybody who gives an interval that represents a false positive -- i.e. an interval that doesn't cover the true value of the heads probability -- will be beheaded.

If we wanted to express the /a posteriori/ probability distribution function of the coin's weight, then of course a credibility interval is what does that. The answer will always be the interval [0.5, 0.5] irrespective of outcome. Even if you flip zero heads or one head, you'll still say [0.5, 0.5] because it's a heck of a lot more probable that the king drew a fair coin and you had a 1/1024 day getting ten heads in a row, than that the king drew the unfair coin.

So this is not a good idea for the courtiers and courtesans to use! Because when the unfair coin is drawn, the whole room (all 100 people) will be wrong and they'll all get beheaded.

In this world where the most important thing is false positives, what we need is an absolute guarantee that the rate of false positives will be less than 5%, no matter which coin is drawn. Then we need to use a confidence interval, like Blyth-Still-Casella or Clopper-Pearson, that works and provides at least 95% coverage irrespective of the true value of the parameter, even in the worst case. If everybody uses this method instead, then no matter which coin is drawn, at the end of the day we can guarantee that the expected number of wrong people will be no more than five.

So the point is: if your criterion requires bounding false positives (or equivalently, guaranteeing coverage), you gotta go with a confidence interval. That's what they do.

(Of course if you also care about false negatives, you'll need a method that makes guarantees about those too...)

Frequentist confidence intervals bound the rate of false positives (Type I errors), and guarantee their coverage will be bounded below by the confidence parameter, even in the worst case. Bayesian credibility intervals don't.

So if the thing you care about is false positives and you need to bound them, confidence intervals are the the approach that you'll want to use.

For example, let's say you have an evil king with a court of 100 courtiers and courtesans and he wants to play a cruel statistical game with them. The king has a bag of a trillion fair coins, plus one unfair coin whose heads probability is 10%. He's going to perform the following game. First, he'll draw a coin uniformly at random from the bag.

Then the coin will be passed around a room of 100 people and each one will be forced to do an experiment on it, privately, and then each person will state a 95% uncertainty interval on what they think the coin's heads probability is.

Anybody who gives an interval that represents a false positive -- i.e. an interval that doesn't cover the true value of the heads probability -- will be beheaded.

If we wanted to express the /a posteriori/ probability distribution function of the coin's weight, then of course a credibility interval is what does that. The answer will always be the interval [0.5, 0.5] irrespective of outcome. Even if you flip zero heads or one head, you'll still say [0.5, 0.5] because it's a heck of a lot more probable that the king drew a fair coin and you had a 1/1024 day getting ten heads in a row, than that the king drew the unfair coin.

So this is not a good idea for the courtiers and courtesans to use! Because when the unfair coin is drawn, the whole room (all 100 people) will be wrong and they'll all get beheaded.

In this world where the most important thing is false positives, what we need is an absolute guarantee that the rate of false positives will be less than 5%, no matter which coin is drawn. Then we need to use a confidence interval, like Blyth-Still-Casella or Clopper-Pearson, that works and provides at least 95% coverage irrespective of the true value of the parameter, even in the worst case. If everybody uses this method instead, then no matter which coin is drawn, at the end of the day we can guarantee that the expected number of wrong people will be no more than five.

So the point is: if your criterion requires bounding false positives (or equivalently, guaranteeing coverage), you gotta go with a confidence interval. That's what they do. Credibility intervals may be a more intuitive way of expressing uncertainty, they may perform pretty well from a frequentist analysis, but they are not going to provide the guaranteed bound on false positives you'll get when you go asking for it.

(Of course if you also care about false negatives, you'll need a method that makes guarantees about those too...)