Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

xkcd comic number 1132

This xkcd comic (Frequentists vs. Bayesians) makes fun of a frequentist statistician who derives an obviously wrong result.

However it seems to me that his reasoning is actually correct in the sense that it follows the standard frequentist methodology.

So my question is "does he correctly apply the frequentist methodology?"

  • If no: what would be a correct frequentist inference in this scenario? How to integrate "prior knowledge" about the sun stability in the frequentist methodology?
  • If yes: wtf? ;-)
share|improve this question
12  
Discussion on Gelman's blog: andrewgelman.com/2012/11/16808 –  Glen Nov 11 '12 at 16:27
3  
I think a lot is wrong, both from the frequentist and Bayesian point of view. My biggest criticism each: First, P values are ultimately heuristics and are properties of a number of things including the statistical problem, data and experiment. Here, all three are grossly misrepresented for that particular question. Second, the "Bayesian" uses a decision theoretic approach which need not be Bayesian. It's funny, though. –  Momo Nov 11 '12 at 17:33
3  
To take it out of the statistics realm....the sun isn't massive enough to go nova. QED, the Bayesian is right. (The Sun will instead become a Red Giant) –  Ben Brocka Nov 12 '12 at 17:15
2  
@Glen et alii, in particular, note Randall Munroe's response to Gelman: andrewgelman.com/2012/11/16808/#comment-109366 –  jthetzel Nov 12 '12 at 17:48
    
I believe the comic confounds estimation and hypothesis testing (Basic mistake!). The machine estimates,using a decision-theoretic approach, the probability of an event. The outcome is 0 or 1, based on the decision rule. The frequentist statistician relates this with a p-value (why? Just for fun). He/she should have related this value with a point estimator. –  user10525 Nov 12 '12 at 18:00
show 1 more comment

11 Answers 11

The main issue is that the first experiment (Sun gone nova) is not repeatable, which makes it highly unsuitable for frequentist methodology that interprets probability as estimate of how frequent an event is giving that we can repeat the experiment many times. In contrast, bayesian probability is interpreted as our degree of belief giving all available prior knowledge, making it suitable for common sense reasoning about one-time events. The dice throw experiment is repeatable, but I find it very unlikely that any frequentist would intentionally ignore the influence of the first experiment and be so confident in significance of the obtained results.

Although it seems that author mocks frequentist reliance on repeatable experiments and their distrust of priors, giving the unsuitability of the experimental setup to the frequentist methodology I would say that real theme of this comic is not frequentist methodology but blind following of unsuitable methodology in general. Whether it's funny or not is up to you (for me it is) but I think it more misleads than clarifies the differences between the two approaches.

share|improve this answer
    
(+1) A nice reference on this strong and crucial assumption of repeatability in frequentism is Statistical Inference in Science (2000), chapter 1. (Although there are so many issues that it is difficult to tell which one is the main one) –  user10525 Nov 12 '12 at 17:41
9  
Not so fast with the repeatability argument... First, the experiment that is repeatable is the querying of the machine not the sun going nova The truth of that is the fixed but unknown object of inference. The querying experiment can certainly be repeated, and if it were for a few more times the frequentist strategy could easily seem reasonable. –  conjugateprior Nov 13 '12 at 9:42
2  
Second, one should not be too stringent on the repeatability business anyway, lest frequentists be stuck not being able to infer anything at all in non-experimental situations. Assume for a moment that 'sun goes nova' was the candidate event. I'm no physicist, but I'm told that the event 'sun goes nova' happens rather often (just not so much around here) so this sounds to me like a repeat. In any case, folk like David Cox (in 'Foundations of Statistics') cheerfully say things like: "the repetitions contemplated are almost always hypothetical. This by itself seems no drawback". –  conjugateprior Nov 13 '12 at 9:51
4  
We could view the sun as a random sample from a population of suns in parallel universes in which we could in principle repeat the experiment if only we had a quantum mirror! ;o) –  Dikran Marsupial Nov 13 '12 at 17:46
add comment

As far as I can see the frequentist bit is reasonable this far:

Let $H_0$ be the hypothesis that the sun has not exploded and $H_1$ be the hypothesis that it has. The p-value is thus the probability of observing the result (the machine saying "yes") under $H_0$. Assuming that the machine correctly detects the presence of absence of neutrinos, then if the machine says "yes" under $H_0$ then it is because the machine is lying to us as a result of rolling two sixes. Thus the p-value is 1/36, so following normal quasi-Fisher scientific practice, a frequentist would reject the null hypothesis, at the 95% level of significance.

But rejecting the null hypothesis does not mean that you are entitled to accept the alternate hypothesis, so the frequentists conclusion is not justified by the analysis. Frequentist hypothesis tests embody the idea of falsificationism (sort of), you can't prove anything is true, only disprove. So if you want to assert $H_1$, you assume $H_0$ is true and only proceed if you can show that $H_0$ is inconsistent with the data. However that doesn't mean $H_1$ is true, just that it survives the test and continues as a viable hypothesis at least as far as the next test.

The Bayesian is also merely common sense, noting that there is nothing to lose by making the bet. I'm sure frequentist approaches, when the false-positive and false-negative costs are taken into account (Neyman-Peason?) would draw the same conclusion as being the best strategy in terms of long-run gain.

To summarise: Both the frequentist and Bayesian are being sloppy here: The frequentist for blindly following a recipe without considering the appropriate level of significance, false-positive/false-negative costs or the physics of the problem (i.e. not using his common sense). The Bayesian is being sloppy for not stating his priors explicitly, but then again using common sense the priors he is using are obviously correct (it is much more likely that the machine is lying than sun actually having exploded), the sloppiness is perhaps excusable.

share|improve this answer
    
You say: rejecting the null hypothesis does not mean that you are entitled to accept the alternate hypothesis, and: that doesn't mean H1 is true. So, what does it means rejecting the null hypothesis that the sun has not gone supernova? –  glassy Jan 17 '13 at 9:23
    
Rejecting the null hypothesis simply means that the observation would be unlikely IF H0 were true. You should not "accept" H1 on this basis as it is basically saying that H1 must be true because the observations would be unlikely if H0 were true. However the observations may also be unlikely under H1 (which the null ritual ignores) and H1 may be less likely than H0 a-priori (which the null ritual also ignores). Accepting hypotheses is a slippery slope towards interpreting a frequentist test as a Bayesian test, which commonly results in misunderstandings in less elementary cases. –  Dikran Marsupial Jan 17 '13 at 10:29
    
In this case it is intuitively reasonable to take this step, but there may be more complex cases where it isn't, and one should not implicitly mix statistical reasoning with intuitive reasoning. Better to err on the side of caution. –  Dikran Marsupial Jan 17 '13 at 10:31
    
Just stumbled upon your comment. And I have the same question that @glassy had. I would like to object to your comment that if your hypotheses cover the entire space of events, here being {"Sun has gone nova", "Sun didn't go nova"}, I have difficulties understanding your point how can rejecting the "Sun has gone nova" not automatically lead to "Sun didn't go nova". Declaring a statement false implies its negation must be true. It would be great if you coul provide some reliable reference text where this point is clearly explained if possible. I'd be interested to find out more about it. –  means-to-meaning Nov 5 '13 at 22:52
    
Rejecting the null hypothesis does not automatically mean that the null hypothesis is probably false, just that it is reasonable to continue with the alternative hypothesis. This is (in part) because the frequentist hypothesis test does not take into account the prior probabilities of the hypotheses. More fundamentally, frequentist methods cannot be used to assign a probability to the truth of any specific hypothesis, so the link between "we can reject the null hypothesis" and "the null hypothesis is probably false" is an entirely subjective one, as far as I can see. –  Dikran Marsupial Nov 6 '13 at 10:08
show 2 more comments

The greatest problem that I see is that there is no test statistic derived. $p$-value (with all the criticisms that Bayesian statisticians mount against it) for a value $t$ of a test statistic $T$ is defined as ${\rm Prob}[T \ge t| H_0]$ (assuming that the null is rejected for greater values of $T$, as would be a case with $\chi^2$ statistics, say). If you need to reach a decision of greater importance, you can increase the critical value and push the rejection region further up. Effectively, that's what multiple testing corrections like Bonferroni do, instructing you to use a much lower threshold for $p$-values. Instead, the frequentist statistician is stuck here with the tests of sizes on the grid of $0, 1/36, 2/36, \ldots$.

Of course, this "frequentist" approach is unscientific, as the result will hardly be reproducible. Once Sun goes supernova, it stays supernova, so the detector should keep saying "Yes" again and again. However, a repeated running of this machine is unlikely to yield the "Yes" result again. This is recognized in areas that want to present themselves as rigorous and try to reproduce their experimental results... which, as far as I understand, happens with probability anywhere between 5% (publishing the original paper was a pure type I error) and somewhere around 30-40% in some medical fields. Meta-analysis folks can fill you in with better numbers, this is just the buzz that comes across me from time to time through the statistics grapevine.

I don't know if this troubles you, but the Bayesian statistician is shown here as the guy who knows no math and has a gambling problem. A proper Bayesian statistician would postulate the prior, discuss its degree of objectivity, derive the posterior, and demonstrate how much they learned from the data. None of that was done, so Bayesian process has been oversimplified just as much as the frequentist one has been.

This situation demonstrates the classical screening for cancer issue (and I am sure biostatisticians can describe it better than I could). When screening for a rare disease with an imperfect instrument, most of the positives come out to be false positives. Smart statisticians know that, and know better to follow up cheap and dirty screeners with more expensive and more accurate biopsies.

share|improve this answer
2  
If I understand your first paragraph correctly, you're saying that the threshold (0.05 in the comic) is set too high. If the comic had five dice instead of two, would you accept the threshold as being low enough? How do you decide the threshold anyway? –  ShreevatsaR Nov 11 '12 at 18:52
8  
I thought the Bayesian statistician simply took into account that the chances of the sun exploding are much, much smaller than the chances of the machine lying (so, not necessarily a clueless gambler). –  josh Nov 11 '12 at 19:09
6  
More to the point: IF the sun goes nova, the winner of the bet will not be able to cash his 50$ ... –  kjetil b halvorsen Nov 11 '12 at 20:14
4  
I think the point here is the frequentist statistician is following a recipe without thinking of the true purpose of the analysis. The so-called "Bayesian" is not actually being a Bayesian, just someone using their common sense. There are plenty of examples of blind recipe following in science journals, which is why the cartoon is amusing. –  Dikran Marsupial Nov 12 '12 at 12:46
2  
Once it is in SAS (for survival models, and may be for some other stuff by now as well) or in AMOS and Mplus (latent variable models), it is a cookbook alright. I would not expect the average users of these latter packages to even understand what Gibbs sampler does. I have seen less sophisticated users to fit a model in say lmer and then use the point estimates and the standard errors as means and standard deviations of their priors... and reanalyze the same data, now in Bayesian way. The cookbooking of anything useful in statistics is largely unstoppable, and NHST is one such result. –  StasK Nov 12 '12 at 18:00
show 5 more comments

Why does this result seem "wrong?" A Bayesian would say that the result seems counter-intuitive because we have "prior" beliefs about when the sun will explode, and the evidence provided by this machine isn't enough to wash out those beliefs (mostly because of it's uncertainty due to the coin flipping). But a frequentist is able to make such an assessment, he simply must do so in the context of data, as opposed to belief.

The real source of the paradox is the fact that the frequentist statistical test performed doesn't take into account all of the data available. There's no problem with the analysis in the comic, but the result seems strange because we know that the sun most likely won't explode for a long time. But HOW do we know this? Because we've made measurements, observations, and simulations that can constrain when the sun will explode. So, our full knowledge should take those measurements and data points into account.

In a Bayesian analysis, this is done by using those measurements to construct a prior (although, the procedure to turn measurements into a prior isn't well-defined: at some point there must be an initial prior, or else it's "turtles all the way down"). So, when the Bayesian uses his prior, he's really taking into account a lot of additional information that the frequentist's p-value analysis isn't privy to.

So, to remain on equal footing, a full frequentist analysis of the problem should include the same additional data about the sun exploding that is used to construct the bayesian prior. But, instead of using priors, a frequentist would simply expand the likelihood that he's using to incorporate those other measurements, and his p-value would be calculated using that full likelihood.

$L = L$(Machine Said Yes | Sun Has Exploded) * $L$(All other data about the sun | Sun Has Exploded)

A full frequentist analysis would most likely show that the second part of the likelihood will be much more constraining and will be the dominant contribution to the p-value calculation (because we have a wealth of information about the sun, and the errors on this information are small (hopefully)).

Practically, one need not go out and collect all data points obtained from the last 500 years to do a frequentist calculation, one can approximate them as some simple likelihood term that encodes the uncertainty as to whether the sun has exploded or not. This will then become similar to the Bayesian's prior, but it is slightly different philosophically because it's a likelihood, meaning that it encodes some previous measurement (as opposed to a prior, which encodes some a priori belief). This new term will become a part of the likelihood and will be used to build confidence intervals (or p-values or whatever), as opposed to the bayesian prior, which is integrated over to form credible intervals or posteriors.

share|improve this answer
    
This should be the accepted or most voted answer. –  user023472 Jul 15 at 23:05
add comment

Now that CERN has decided that neutrinos are not faster than light - the electromagnetic radiation shock front would hit the earth before the neutrino change was noticed. This would have at the least (in the very short term) spectacular auroral effects. Thus the fact that it is dark would not prevent the skies from being lit up; the moon from shining excessively brightly (cf Larry Niven's "Inconstant Moon") and spectacular flashes as artificial satellites were vapourised and self combusted.

All in all - perhaps the wrong test? (And whilst there may have been prior - there would be insufficient time for a realistic determination of posterior.

share|improve this answer
1  
All the more reason to reject the hypothesis that the sun has exploded, then. :-) –  ShreevatsaR Nov 13 '12 at 4:46
    
So this is what is meant at the end of the article when the authors say: "confirmatory studies are needed"? –  DWin Nov 30 '13 at 21:28
    
Actually, casually revisiting this the clear inference is in the title. The machine detects whether the sun has gone nova. There is no chance of error in the detection. The neutrino bit is irrelevant. Given that, then the statistics are such that the machine will reply "no", "no", "no"... with a 1/36 chance of being a false statement (yes) until a one off event which terminates the statistical process occurs - this will also have a 1/36 chance of being falsely reported (no), if the machine is queried during the 8 odd minute interval it takes to become evident on earth. –  SimonN Dec 1 '13 at 8:31
add comment

There's nothing wrong with this comic, and the reason has nothing to do with statistics. It's economics. If the frequentist is correct, the Earth will be tantamount to uninhabitable within 48 hours. The value of \$50 will be effectively null. The Bayesian, recognizing this, can make the bet knowing that his benefit is \$50 in the normal case, and marginally nothing in the sun-exploded case.

share|improve this answer
add comment

I agree with @GeorgeLewis that it may be premature to conclude the Frequentist approach is wrong - let's just rerun the neutrino detector several more times to collect more data. No need to mess around with priors.

share|improve this answer
add comment

A simpler point that may be lost among all the verbose answers here is that the frequentist is depicted drawing his conclusion based upon a single sample. In practice you would never do this.

Reaching a valid conclusion requires a statistically significant sample size (or in other words, science needs to be repeatable). So in practice the frequentist would run the machine multiple times and then come to a conclusion about the resulting data.

Presumably this would entail asking the machine the same question several more times. And presumably if the machine is only wrong 1 out of every 36 times a clear pattern will emerge. And from that pattern (rather then from one single reading) the frequentist will draw a (fairly accurate, I would say) conclusion regarding whether or not the sun has exploded.

share|improve this answer
2  
What do you mean by "statistically significant sample size"? –  Momo Nov 16 '12 at 9:08
    
@Momo - More than a single sample, that's for sure. It's not valid to observe an improbable outcome and then make conclusions that the improbable has happened without first repeating the observation to make sure it wasn't a fluke. If you want an exact number that represents a statistically significant sample size or an algorithm to determine an exact number, probably a statistician can provide one; but I'm not a statistician. –  aroth Nov 19 '12 at 7:30
2  
I don't think there is a particular problem with having a sample size of 1 is the issue, the problem is that the test has no statistical power (i.e. the test will never reject the null hypothesis when it is false). However, this reveals a problem with the "null ritual" being lampooned in the article, which ignores the issue of statistical power (and what H1 actually is, or prior information relevant to the problem). –  Dikran Marsupial Nov 30 '13 at 11:44
add comment

The answer for your question: "does he correctly apply the frequentist methodology?" is no, he does not applied precisely the frequentist approach. The p-value for this problem is not exactly 1/36.

We first must note that the involved hypotheses are

H0: The Sun has not exploded,

H1: The Sun has exploded.

Then,

p-value = P("the machine returns yes" | the Sun hasn't exploded).

To compute this probability, we must note that "the machine returns yes" is equivalent to "the neutrino detector measures the Sun exploding AND tells the true result OR the neutrino detector does not measure the Sun exploding AND lies to us".

Assuming that the dice throwing is independent of the neutrino detector measurement, we can compute the p-value by defining:

p0 = P("the neutrino detector measures the Sun exploding" |the Sun hasn't exploded),

Then, the p-value is

p-value = p0 x 35/36 + (1-p0) x 1/36 = (1/36) x (1+ 34 x p0).

For this problem, the p-value is a number between 1/36 and 35/36. The p-value is equal 1/36 if and only if p0=0. That is, a hidden assumption in this cartoon is that the detector machine will never measure the Sun exploding if the Sun hasn't exploded.

Moreover, much more information should be inserted in the likelihood about external evidences of an anova explosion going on.

All the Best.

share|improve this answer
add comment

I don't see any problem with the frequentist's approach. If the null hypothesis is rejected, the p-value is the probability of a type 1 error. A type 1 error is rejecting a true null hypothesis. In this case we have a p-value of 0.028. This means that among all the hypothesis tests with this p-value ever conducted, roughly 3 out of a hundred will reject a true null hypothesis. By construction, this would be one of those cases. Frequentists accept that sometimes they'll reject true null hypothesis or retain false null hypothesis (Type 2 errors), they've never claimed otherwise. Moreover, they precisely quantify the frequency of their erroneous inferences in the long run.

Perhaps, a less confusing way of looking at this result is to exchange the roles of the hypotheses. Since the two hypotheses are simple, this is easy to do. If the null is that the sun went nova, then the p-value is 35/36=0.972. This means that this is no evidence against the hypothesis that the sun went nova, so we can't reject it based on this result. This seems more reasonable. If you are thinking. Why would anybody assume the sun went nova? I would ask you. Why would anybody carry out such an experiment if the very thought of the sun exploding seems ridiculous?

I think this just shows that one has to assess the usefulness of an experiment beforehand. This experiment, for example, would be completely useless because it tests something we already know simply from looking up to the sky (Which I'm sure produces a p-value that is effectively zero). Designing a good experiment is a requirement to produce good science. If your experiment is poorly designed, then no matter what statistical inference tool you use, your results are unlikely to be useful.

share|improve this answer
    
Certainly, but the Bayesian can still infer a reasonable conclusion with the given data / experiment results. Sometimes you can't repeat an experiment or design it the way you want. –  user023472 Jul 15 at 23:12
add comment

How to integrate "prior knowledge" about the sun stability in the frequentist methodology?

Very interesting topic.

Here are just some thoughts, not a perfect analysis...

Using the Bayesian approach with a noninformative prior typically provides a statistical inference comparable to the frequentist one.

Why does the Bayesian has a strong prior belief that the sun has not exploded ? Because he knows as everyone that the sun has never exploded since its beginning.

We can see on some simple statistical models with conjugate priors that using a prior distribution is equivalent to use the posterior distribution derived from a noninfomative prior and preliminary experiments.

The sentence above suggests that the Frequentist should conclude as the Bayesian by including the results of preliminary experiments in his model. And this is what the Bayesian actually does: his prior comes from his knowledge of the preliminary experiments !

Let $N$ be the age of the sun in days, and $x_i$ be the status of the sun (0 = exploded / 1 = not exploded) at day $i$. Assume the $x_i$ are i.i.d Bernoulli variates with probability of succes $\theta$. The realizations of the $x_i$ have been observed : $x_i=1$ for all $i =1,\ldots,N$.

In the current problem, we have $N+1$ observations : the $x_i$ and the result $y=\{\text{Yes}\}$ of the detector. The natural question is : what is the probability that the sun has exploded, that is, what is $\Pr(x_{N+1}=0)$ ? This is $\theta$ and estimating $\theta$ from the available observations $x_1, \ldots, x_N$ and $y$ yields an estimate highly close to $1$ because $N$ is huge, and the "unexpected" value $y=\{\text{Yes}\}$ has a negligible impact on the estimate of $\theta$. And the Bayesian intends to reflect this information through his prior distribution about $\theta$.

From this perspective I don't see how to rephrase the question in terms of hypothesis testing. Taking $H_0 =\{\text{the sun has not exploded}\}$ makes no sense cause it is a possible issue of the experiment in my interpretation, not a true/false hypothesis. Maybe this is the error of the Frequentist ?

share|improve this answer
    
The passage "... he knows as everyone that the sun has never exploded since its beginning" brings to mind a story about a recent American holiday in which millions of turkeys (Meleagris gallopavo) are consumed. As time goes on, each day any intelligent turkey "knows as everyone" that she will be fed and cared for, right until that fateful (and wholly unexpected--to her) day in the middle of November! Similarly, our confidence in the stability of the sun ought to be low if all we had to rely on was the relatively short history of human observation of it. –  whuber Nov 30 '13 at 19:06
    
@whuber I would have preferred to send you this message privately. Is there a connection between your comment and the topic of discussion? I do not know if it's me that makes me ideas, but it's been several times that I feel that you comment on my answers mainly to say something against my answers. The exercise posed by the OP is the interpretation of a cartoon, and I feel that you criticize my answer as if I were talking about a real issue. Recently I did not appreciate and I still have not understood why you evoked a likely "intent" behind my answers. –  Stéphane Laurent Nov 30 '13 at 19:26
    
There was no criticism, implied or intended: sometimes a comment is really just ... a comment. It tried to highlight (in a way intended to be humorous) important questions hinted at but not addressed in your answer. I am sorry that you perceive this as either personal or an attack. BTW, this is a real question: it asks How to integrate "prior knowledge" ... in the frequentist methodology? This question evokes Hume's criticism of inductive inference and goes to issues in the philosophy of science as well as the very foundations of statistics. It is well worth some careful thought! –  whuber Nov 30 '13 at 19:40
    
It might be worth pointing out, too, that a substantial proportion of your reputation is due to my votes for your answers--which I offer as material evidence that there is no systematic behavior on my part against you. –  whuber Nov 30 '13 at 19:49
1  
No, I have understand your comment. The French Google translation of your comment is already strange, but by combining my skills in English and the strange Google translations, I'm able to get a correct translation. I will be more relax next month, likely. –  Stéphane Laurent Nov 30 '13 at 20:40
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.