Timeline for Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals
Current License: CC BY-SA 4.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| S Apr 14 at 0:01 | history | suggested | feetwet | CC BY-SA 4.0 |
Copy edited
|
| Apr 13 at 23:46 | review | Suggested edits | |||
| S Apr 14 at 0:01 | |||||
| Jul 4, 2022 at 16:37 | comment | added | fblundun | You could have made the example simpler: just suppose the king asks the courtesans to show their working, and executes anybody who used Bayes' Theorem. A clear case where frequentist methods perform better! | |
| Feb 5, 2011 at 11:25 | comment | added | probabilityislogic | @Keith - if what you say is true, then you should point out the mistake I have made in my answer (relating to Wasserman's example). Because the CI in that case does not have the 95% coverage for all values of the parameter. So if you are correct, then logically, I must have made a mistake somewhere in the calculations. | |
| Feb 2, 2011 at 5:37 | comment | added | Keith Winstein | Hi -- yes, a confidence interval's coverage probability is bounded below by the confidence parameter. So a 95% confidence interval will have coverage of at least 95%, irrespective of the true value of the parameter. A credibility interval does not make this guarantee, and can have coverage lower than its probability -- it can even have 0% coverage for some values of the parameter, as in the "king" example. See stats.stackexchange.com/questions/2272/… for a fuller explanation. | |
| Jan 31, 2011 at 7:11 | comment | added | probabilityislogic | A confidence interval does not bound the rate of false positives - see my answer below for a counter-example to back up my claim. | |
| Jan 27, 2011 at 14:04 | comment | added | probabilityislogic | I would have thought the above argument holds just as valid for the frequentist as well. The argument above (as far as I can tell) does not invoke any specifically Bayesian or Frequentist principles (although it does invoke the principle of sanity). | |
| Jan 27, 2011 at 13:24 | comment | added | probabilityislogic | The question as posed is a bit ambiguous, because it does not stated clearly what information the 100 people have. Do they know the distribution in the bag? for if they do, they "experiment" is useless, one would just give the interval $[0.1,0.5]$ or even just the two values $0.1$ and $0.5$ (does give required $\text{100%} \geq \text{95%}$ coverage). If we only know that there are a bag of coins to be drawn from, the Bayesian would specify the whole [0,1] interval, because false positives is all that matters in this question (and the size of the interval does not). | |
| Jan 19, 2011 at 6:18 | comment | added | probabilityislogic | Confidence intervals, in my opinion are completely and utterly useless UNLESS the experiment is to be repeated a moderate number of times (10 or more). Because whether or not an $\alpha$ level CI contains the true parameter is basically a $Bernouli(\alpha)$ random variable which has been "mixed up" so that we don't know whether we have observed a "success" or a "failure". Also this problem it is impossible to give an "exact" CI, because $1^{12}$ times its 0.5 and 1 time its 0.1. Show me 95% of this set? it doesn't exist! Wouldn't you just give the set of two numbers {0.5,0.1}? | |
| Sep 7, 2010 at 19:57 | comment | added | Dikran Marsupial | The confidence interval only provides a bound on the expected number of false positives, it is not possible to put an absolute bound on the number of false positives for a particular sample (neglecting a trivial interval of [0,1]). A Bayesian would determine an interval such that the probability of of more than five beheadings is less than some threshold value (e.g. 10^-6). This seems at least as useful as a bound on the expected number of beheadings and has the advantage of being a (probabilistic) bound on what happens to the actual sample of courtiers. I'd say this one was a clear draw. | |
| Sep 7, 2010 at 19:47 | comment | added | Dikran Marsupial | I agree with the first point, it is a matter of "horses for courses", but examples which show where the boundaries lie are interesting and provide insight into the "courses" best suited to each "horse". However, the examples must be fair, so that the criterion for success matches the question as posed (Jaynes is perhaps not completely immune to that criticism, which I will address in my answer which I will post later). | |
| Sep 7, 2010 at 4:55 | comment | added | Keith Winstein | Dikran, there needn't be "Bayesians" and "Frequentists." They're not incompatible schools of philosophy to which one may subscribe to only one! They are mathematical tools whose efficacy can be demonstrated in the common framework of probability theory. My point is that IF the requirement is an absolute bound on false positives no matter the true value of the parameter, THEN a confidence interval is the method that accomplishes that. Of course we all agree on the same axioms of probability and the same answer can be derived many ways. | |
| Sep 6, 2010 at 8:29 | comment | added | Dikran Marsupial | Having though about this a bit more, this example is invalid as the criterion used to measure success is not the same as that implied by the question posed by the king. The problem is in the "no matter which coin is drawn", a clause that is designed to trip up any method that uses the prior knowledge about the rarity of the biased coin. As it happens, Bayesains can derive bounds as well (e.g. PAC bounds) and if asked would have done so, and I suspect the answer would be the same as the Clopper-Pearson interval. To be a fair test, the same information must be given to both approaches. | |
| Sep 4, 2010 at 9:57 | comment | added | Dikran Marsupial | Note also that if the selected coin is flipped often enough, then eventually the Bayesian confidence interval will be centered on the long run frequency of heads for the particular coin rather than on the prior. If my life depended on the interval containing the true probability of a head I wouldn't flip the coin just once! | |
| Sep 4, 2010 at 9:10 | comment | added | Dikran Marsupial | Food for thought, however the particular example is unfair as the frequentist approach is allowed to consider the relative costs of false-positive and false-negative costs, but the Bayesian approach isn't. The correct thing to do according to Bayesian decision theory is to give an interval of [0,1] as there is no penalty associated with false-negatives. Thus in a like-for-like comparison of frameworks, none of the Bayesians would ever be beheaded either. The issue about bounding false-positives though gives me a direction in which to look for an answer to Jaynes' challenge. | |
| Sep 4, 2010 at 5:26 | history | edited | Keith Winstein | CC BY-SA 2.5 |
added 247 characters in body
|
| Sep 4, 2010 at 4:22 | history | answered | Keith Winstein | CC BY-SA 2.5 |