What's wrong with XKCD's Frequentists vs. Bayesians comic?

Question

This xkcd comic (Frequentists vs. Bayesians) makes fun of a frequentist statistician who derives an obviously wrong result.

However it seems to me that his reasoning is actually correct in the sense that it follows the standard frequentist methodology.

So my question is "does he correctly apply the frequentist methodology?"

• If no: what would be a correct frequentist inference in this scenario? How to integrate "prior knowledge" about the sun stability in the frequentist methodology?
• If yes: wtf? ;-)

Answer 1

The main issue is that the first experiment (Sun gone nova) is not repeatable, which makes it highly unsuitable for frequentist methodology that interprets probability as estimate of how frequent an event is giving that we can repeat the experiment many times. In contrast, bayesian probability is interpreted as our degree of belief giving all available prior knowledge, making it suitable for common sense reasoning about one-time events. The dice throw experiment is repeatable, but I find it very unlikely that any frequentist would intentionally ignore the influence of the first experiment and be so confident in significance of the obtained results.

Although it seems that author mocks frequentist reliance on repeatable experiments and their distrust of priors, giving the unsuitability of the experimental setup to the frequentist methodology I would say that real theme of this comic is not frequentist methodology but blind following of unsuitable methodology in general. Whether it's funny or not is up to you (for me it is) but I think it more misleads than clarifies the differences between the two approaches.

Answer 2

Why does this result seem "wrong?" A Bayesian would say that the result seems counter-intuitive because we have "prior" beliefs about when the sun will explode, and the evidence provided by this machine isn't enough to wash out those beliefs (mostly because of it's uncertainty due to the coin flipping). But a frequentist is able to make such an assessment, he simply must do so in the context of data, as opposed to belief.

The real source of the paradox is the fact that the frequentist statistical test performed doesn't take into account all of the data available. There's no problem with the analysis in the comic, but the result seems strange because we know that the sun most likely won't explode for a long time. But HOW do we know this? Because we've made measurements, observations, and simulations that can constrain when the sun will explode. So, our full knowledge should take those measurements and data points into account.

In a Bayesian analysis, this is done by using those measurements to construct a prior (although, the procedure to turn measurements into a prior isn't well-defined: at some point there must be an initial prior, or else it's "turtles all the way down"). So, when the Bayesian uses his prior, he's really taking into account a lot of additional information that the frequentist's p-value analysis isn't privy to.

So, to remain on equal footing, a full frequentist analysis of the problem should include the same additional data about the sun exploding that is used to construct the bayesian prior. But, instead of using priors, a frequentist would simply expand the likelihood that he's using to incorporate those other measurements, and his p-value would be calculated using that full likelihood.

$L = L$(Machine Said Yes | Sun Has Exploded) * $L$(All other data about the sun | Sun Has Exploded)

A full frequentist analysis would most likely show that the second part of the likelihood will be much more constraining and will be the dominant contribution to the p-value calculation (because we have a wealth of information about the sun, and the errors on this information are small (hopefully)).

Practically, one need not go out and collect all data points obtained from the last 500 years to do a frequentist calculation, one can approximate them as some simple likelihood term that encodes the uncertainty as to whether the sun has exploded or not. This will then become similar to the Bayesian's prior, but it is slightly different philosophically because it's a likelihood, meaning that it encodes some previous measurement (as opposed to a prior, which encodes some a priori belief). This new term will become a part of the likelihood and will be used to build confidence intervals (or p-values or whatever), as opposed to the bayesian prior, which is integrated over to form credible intervals or posteriors.

Answer 3

As far as I can see the frequentist bit is reasonable this far:

Let $H_0$ be the hypothesis that the sun has not exploded and $H_1$ be the hypothesis that it has. The p-value is thus the probability of observing the result (the machine saying "yes") under $H_0$. Assuming that the machine correctly detects the presence of absence of neutrinos, then if the machine says "yes" under $H_0$ then it is because the machine is lying to us as a result of rolling two sixes. Thus the p-value is 1/36, so following normal quasi-Fisher scientific practice, a frequentist would reject the null hypothesis, at the 95% level of significance.

But rejecting the null hypothesis does not mean that you are entitled to accept the alternate hypothesis, so the frequentists conclusion is not justified by the analysis. Frequentist hypothesis tests embody the idea of falsificationism (sort of), you can't prove anything is true, only disprove. So if you want to assert $H_1$, you assume $H_0$ is true and only proceed if you can show that $H_0$ is inconsistent with the data. However that doesn't mean $H_1$ is true, just that it survives the test and continues as a viable hypothesis at least as far as the next test.

The Bayesian is also merely common sense, noting that there is nothing to lose by making the bet. I'm sure frequentist approaches, when the false-positive and false-negative costs are taken into account (Neyman-Peason?) would draw the same conclusion as being the best strategy in terms of long-run gain.

To summarise: Both the frequentist and Bayesian are being sloppy here: The frequentist for blindly following a recipe without considering the appropriate level of significance, false-positive/false-negative costs or the physics of the problem (i.e. not using his common sense). The Bayesian is being sloppy for not stating his priors explicitly, but then again using common sense the priors he is using are obviously correct (it is much more likely that the machine is lying than sun actually having exploded), the sloppiness is perhaps excusable.

Answer 4

The greatest problem that I see is that there is no test statistic derived. $p$-value (with all the criticisms that Bayesian statisticians mount against it) for a value $t$ of a test statistic $T$ is defined as ${\rm Prob}[T \ge t| H_0]$ (assuming that the null is rejected for greater values of $T$, as would be a case with $\chi^2$ statistics, say). If you need to reach a decision of greater importance, you can increase the critical value and push the rejection region further up. Effectively, that's what multiple testing corrections like Bonferroni do, instructing you to use a much lower threshold for $p$-values. Instead, the frequentist statistician is stuck here with the tests of sizes on the grid of $0, 1/36, 2/36, \ldots$.

Of course, this "frequentist" approach is unscientific, as the result will hardly be reproducible. Once Sun goes supernova, it stays supernova, so the detector should keep saying "Yes" again and again. However, a repeated running of this machine is unlikely to yield the "Yes" result again. This is recognized in areas that want to present themselves as rigorous and try to reproduce their experimental results... which, as far as I understand, happens with probability anywhere between 5% (publishing the original paper was a pure type I error) and somewhere around 30-40% in some medical fields. Meta-analysis folks can fill you in with better numbers, this is just the buzz that comes across me from time to time through the statistics grapevine.

One other problem from the "proper" frequentist perspective is that rolling a die is the least powerful test, with power = significance level (if not lower; 2.7% power for the 5% significance level is nothing to boast about). Neyman-Pearson theory for t-tests agonizes over demonstrating that this is a UMPT, and a lot of high brow statistical theory (which I barely understand, I have to admit) is devoted to deriving the power curves and finding the conditions when a given test is the most powerful one in a given class. (Credits: @Dikran Marsupial mentioned the issue of power in one of the comments.)

I don't know if this troubles you, but the Bayesian statistician is shown here as the guy who knows no math and has a gambling problem. A proper Bayesian statistician would postulate the prior, discuss its degree of objectivity, derive the posterior, and demonstrate how much they learned from the data. None of that was done, so Bayesian process has been oversimplified just as much as the frequentist one has been.

This situation demonstrates the classical screening for cancer issue (and I am sure biostatisticians can describe it better than I could). When screening for a rare disease with an imperfect instrument, most of the positives come out to be false positives. Smart statisticians know that, and know better to follow up cheap and dirty screeners with more expensive and more accurate biopsies.

Answer 5

I agree with @GeorgeLewis that it may be premature to conclude the Frequentist approach is wrong - let's just rerun the neutrino detector several more times to collect more data. No need to mess around with priors.

Answer 6

There's nothing wrong with this comic, and the reason has nothing to do with statistics. It's economics. If the frequentist is correct, the Earth will be tantamount to uninhabitable within 48 hours. The value of \$50 will be effectively null. The Bayesian, recognizing this, can make the bet knowing that his benefit is \$50 in the normal case, and marginally nothing in the sun-exploded case.

Answer 7

Now that CERN has decided that neutrinos are not faster than light - the electromagnetic radiation shock front would hit the earth before the neutrino change was noticed. This would have at the least (in the very short term) spectacular auroral effects. Thus the fact that it is dark would not prevent the skies from being lit up; the moon from shining excessively brightly (cf Larry Niven's "Inconstant Moon") and spectacular flashes as artificial satellites were vapourised and self combusted.

All in all - perhaps the wrong test? (And whilst there may have been prior - there would be insufficient time for a realistic determination of posterior.

Answer 8

The answer for your question: "does he correctly apply the frequentist methodology?" is no, he does not applied precisely the frequentist approach. The p-value for this problem is not exactly 1/36.

We first must note that the involved hypotheses are

H0: The Sun has not exploded,

H1: The Sun has exploded.

Then,

p-value = P("the machine returns yes" | the Sun hasn't exploded).

To compute this probability, we must note that "the machine returns yes" is equivalent to "the neutrino detector measures the Sun exploding AND tells the true result OR the neutrino detector does not measure the Sun exploding AND lies to us".

Assuming that the dice throwing is independent of the neutrino detector measurement, we can compute the p-value by defining:

p0 = P("the neutrino detector measures the Sun exploding" |the Sun hasn't exploded),

Then, the p-value is

p-value = p0 x 35/36 + (1-p0) x 1/36 = (1/36) x (1+ 34 x p0).

For this problem, the p-value is a number between 1/36 and 35/36. The p-value is equal 1/36 if and only if p0=0. That is, a hidden assumption in this cartoon is that the detector machine will never measure the Sun exploding if the Sun hasn't exploded.

Moreover, much more information should be inserted in the likelihood about external evidences of an anova explosion going on.

All the Best.

Answer 9

A simpler point that may be lost among all the verbose answers here is that the frequentist is depicted drawing his conclusion based upon a single sample. In practice you would never do this.

Reaching a valid conclusion requires a statistically significant sample size (or in other words, science needs to be repeatable). So in practice the frequentist would run the machine multiple times and then come to a conclusion about the resulting data.

Presumably this would entail asking the machine the same question several more times. And presumably if the machine is only wrong 1 out of every 36 times a clear pattern will emerge. And from that pattern (rather then from one single reading) the frequentist will draw a (fairly accurate, I would say) conclusion regarding whether or not the sun has exploded.

Answer 10

This is of course a frequentist 0.05 level test - the null hypothesis is rejected less than 5% of the time under the null hypothesis and even the power under the alternative is great.

On the other hand prior information tells us that the sun going supernova at a by particular point in time is pretty unlikely, but that getting a lie by chance is more likely.

Bottom line: there's not really anything wrong with the comic and it shows that testing implausible hypotheses leads to a high false discovery rate. Additionally, you probably want to take prior information into account in your assessment of offered bets - that's why a Bayesian posterior in combination with decision analysis is so popular.

Answer 11

I don't see any problem with the frequentist's approach. If the null hypothesis is rejected, the p-value is the probability of a type 1 error. A type 1 error is rejecting a true null hypothesis. In this case we have a p-value of 0.028. This means that among all the hypothesis tests with this p-value ever conducted, roughly 3 out of a hundred will reject a true null hypothesis. By construction, this would be one of those cases. Frequentists accept that sometimes they'll reject true null hypothesis or retain false null hypothesis (Type 2 errors), they've never claimed otherwise. Moreover, they precisely quantify the frequency of their erroneous inferences in the long run.

Perhaps, a less confusing way of looking at this result is to exchange the roles of the hypotheses. Since the two hypotheses are simple, this is easy to do. If the null is that the sun went nova, then the p-value is 35/36=0.972. This means that this is no evidence against the hypothesis that the sun went nova, so we can't reject it based on this result. This seems more reasonable. If you are thinking. Why would anybody assume the sun went nova? I would ask you. Why would anybody carry out such an experiment if the very thought of the sun exploding seems ridiculous?

I think this just shows that one has to assess the usefulness of an experiment beforehand. This experiment, for example, would be completely useless because it tests something we already know simply from looking up to the sky (Which I'm sure produces a p-value that is effectively zero). Designing a good experiment is a requirement to produce good science. If your experiment is poorly designed, then no matter what statistical inference tool you use, your results are unlikely to be useful.

Answer 12

How to integrate "prior knowledge" about the sun stability in the frequentist methodology?

Very interesting topic.

Here are just some thoughts, not a perfect analysis...

Using the Bayesian approach with a noninformative prior typically provides a statistical inference comparable to the frequentist one.

Why does the Bayesian has a strong prior belief that the sun has not exploded ? Because he knows as everyone that the sun has never exploded since its beginning.

We can see on some simple statistical models with conjugate priors that using a prior distribution is equivalent to use the posterior distribution derived from a noninfomative prior and preliminary experiments.

The sentence above suggests that the Frequentist should conclude as the Bayesian by including the results of preliminary experiments in his model. And this is what the Bayesian actually does: his prior comes from his knowledge of the preliminary experiments !

Let $N$ be the age of the sun in days, and $x_i$ be the status of the sun (0 = exploded / 1 = not exploded) at day $i$. Assume the $x_i$ are i.i.d Bernoulli variates with probability of succes $\theta$. The realizations of the $x_i$ have been observed : $x_i=1$ for all $i =1,\ldots,N$.

In the current problem, we have $N+1$ observations : the $x_i$ and the result $y=\{\text{Yes}\}$ of the detector. The natural question is : what is the probability that the sun has exploded, that is, what is $\Pr(x_{N+1}=0)$ ? This is $\theta$ and estimating $\theta$ from the available observations $x_1, \ldots, x_N$ and $y$ yields an estimate highly close to $1$ because $N$ is huge, and the "unexpected" value $y=\{\text{Yes}\}$ has a negligible impact on the estimate of $\theta$. And the Bayesian intends to reflect this information through his prior distribution about $\theta$.

From this perspective I don't see how to rephrase the question in terms of hypothesis testing. Taking $H_0 =\{\text{the sun has not exploded}\}$ makes no sense cause it is a possible issue of the experiment in my interpretation, not a true/false hypothesis. Maybe this is the error of the Frequentist ?

Answer 13

If the frequentist is set on using a p-value to determine whether the sun has exploded, his mistake is that he should be testing at a much lower significance level $\alpha$ than 0.05, since the claim that the sun has gone nova and that the frequentist is still alive despite photons from the nova already reaching Earth is so unlikely.

In fact, $\alpha$ should be so small that even if the machine is designed to always tell the truth, the frequentist still doesn't reject the null hypothesis $H_0$ that the sun hasn't gone nova.

Let $q$ be the probability that the frequentist hallucinates that the machine reports that the sun has gone nova. $\alpha$ should be less then $q$. $H_0$ should only be rejected if the probability of the observation under $H_0$ is less than $\alpha$, but this probability is bounded below by $q$, so $H_0$ cannot be rejected. ($q$ could also have been defined as the probability that the machine is broken and always reports that the sun has gone nova.)

Answer 14

In my view, a more correct frequentist analysis would be as follows: H0: The sun has exploded and the machine is telling the truth. H1: The sun has not exploded and the machine is lying.

The p value here is = P(sun exploded) . p(machine is telling the truth) = 0.97 . P(sun exploded)

The statistician can not conclude anything without knowing the nature of the second probability.

Although we know that P(sun exploded) is 0, because sun like stars do not explode into supernovae.