# What's wrong with XKCD's Frequentists vs. Bayesians comic?

This xkcd comic (Frequentists vs. Bayesians) makes fun of a frequentist statistician who derives an obviously wrong result.

However it seems to me that his reasoning is actually correct in the sense that it follows the standard frequentist methodology.

So my question is "does he correctly apply the frequentist methodology?"

• If no: what would be a correct frequentist inference in this scenario? How to integrate "prior knowledge" about the sun stability in the frequentist methodology?
• If yes: wtf? ;-)
• Discussion on Gelman's blog: andrewgelman.com/2012/11/16808
– Glen
Commented Nov 11, 2012 at 16:27
• I think a lot is wrong, both from the frequentist and Bayesian point of view. My biggest criticism each: First, P values are ultimately heuristics and are properties of a number of things including the statistical problem, data and experiment. Here, all three are grossly misrepresented for that particular question. Second, the "Bayesian" uses a decision theoretic approach which need not be Bayesian. It's funny, though.
– Momo
Commented Nov 11, 2012 at 17:33
• To take it out of the statistics realm....the sun isn't massive enough to go nova. QED, the Bayesian is right. (The Sun will instead become a Red Giant) Commented Nov 12, 2012 at 17:15
• @Glen et alii, in particular, note Randall Munroe's response to Gelman: andrewgelman.com/2012/11/16808/#comment-109366 Commented Nov 12, 2012 at 17:48
• The reason the frequentist statistician here is stupid isn't because he is a frequentist, but because he obviously knows how the machine works, therefore knows it is an inappropriate measurement -- and makes an inference anyway. Commented Jun 2, 2015 at 19:08

The main issue is that the first experiment (Sun gone nova) is not repeatable, which makes it highly unsuitable for frequentist methodology that interprets probability as estimate of how frequent an event is giving that we can repeat the experiment many times. In contrast, bayesian probability is interpreted as our degree of belief giving all available prior knowledge, making it suitable for common sense reasoning about one-time events. The dice throw experiment is repeatable, but I find it very unlikely that any frequentist would intentionally ignore the influence of the first experiment and be so confident in significance of the obtained results.

Although it seems that author mocks frequentist reliance on repeatable experiments and their distrust of priors, giving the unsuitability of the experimental setup to the frequentist methodology I would say that real theme of this comic is not frequentist methodology but blind following of unsuitable methodology in general. Whether it's funny or not is up to you (for me it is) but I think it more misleads than clarifies the differences between the two approaches.

• (+1) A nice reference on this strong and crucial assumption of repeatability in frequentism is Statistical Inference in Science (2000), chapter 1. (Although there are so many issues that it is difficult to tell which one is the main one)
– user10525
Commented Nov 12, 2012 at 17:41
• Not so fast with the repeatability argument... First, the experiment that is repeatable is the querying of the machine not the sun going nova The truth of that is the fixed but unknown object of inference. The querying experiment can certainly be repeated, and if it were for a few more times the frequentist strategy could easily seem reasonable. Commented Nov 13, 2012 at 9:42
• Second, one should not be too stringent on the repeatability business anyway, lest frequentists be stuck not being able to infer anything at all in non-experimental situations. Assume for a moment that 'sun goes nova' was the candidate event. I'm no physicist, but I'm told that the event 'sun goes nova' happens rather often (just not so much around here) so this sounds to me like a repeat. In any case, folk like David Cox (in 'Foundations of Statistics') cheerfully say things like: "the repetitions contemplated are almost always hypothetical. This by itself seems no drawback". Commented Nov 13, 2012 at 9:51
• We could view the sun as a random sample from a population of suns in parallel universes in which we could in principle repeat the experiment if only we had a quantum mirror! ;o) Commented Nov 13, 2012 at 17:46
• Why isn't the checking for the sun exploding repeatable? I check every morning, and it hasn't blown up yet.
– GKFX
Commented Aug 16, 2015 at 12:58

Why does this result seem "wrong?" A Bayesian would say that the result seems counter-intuitive because we have "prior" beliefs about when the sun will explode, and the evidence provided by this machine isn't enough to wash out those beliefs (mostly because of it's uncertainty due to the coin flipping). But a frequentist is able to make such an assessment, he simply must do so in the context of data, as opposed to belief.

The real source of the paradox is the fact that the frequentist statistical test performed doesn't take into account all of the data available. There's no problem with the analysis in the comic, but the result seems strange because we know that the sun most likely won't explode for a long time. But HOW do we know this? Because we've made measurements, observations, and simulations that can constrain when the sun will explode. So, our full knowledge should take those measurements and data points into account.

In a Bayesian analysis, this is done by using those measurements to construct a prior (although, the procedure to turn measurements into a prior isn't well-defined: at some point there must be an initial prior, or else it's "turtles all the way down"). So, when the Bayesian uses his prior, he's really taking into account a lot of additional information that the frequentist's p-value analysis isn't privy to.

So, to remain on equal footing, a full frequentist analysis of the problem should include the same additional data about the sun exploding that is used to construct the bayesian prior. But, instead of using priors, a frequentist would simply expand the likelihood that he's using to incorporate those other measurements, and his p-value would be calculated using that full likelihood.

$L = L$(Machine Said Yes | Sun Has Exploded) * $L$(All other data about the sun | Sun Has Exploded)

A full frequentist analysis would most likely show that the second part of the likelihood will be much more constraining and will be the dominant contribution to the p-value calculation (because we have a wealth of information about the sun, and the errors on this information are small (hopefully)).

Practically, one need not go out and collect all data points obtained from the last 500 years to do a frequentist calculation, one can approximate them as some simple likelihood term that encodes the uncertainty as to whether the sun has exploded or not. This will then become similar to the Bayesian's prior, but it is slightly different philosophically because it's a likelihood, meaning that it encodes some previous measurement (as opposed to a prior, which encodes some a priori belief). This new term will become a part of the likelihood and will be used to build confidence intervals (or p-values or whatever), as opposed to the bayesian prior, which is integrated over to form credible intervals or posteriors.

• This should be the accepted or most voted answer. Commented Jul 15, 2014 at 23:05
• My favorite part is "...or else it's 'turtles all the way down.'" Commented Aug 29, 2021 at 15:55

As far as I can see the frequentist bit is reasonable this far:

Let $H_0$ be the hypothesis that the sun has not exploded and $H_1$ be the hypothesis that it has. The p-value is thus the probability of observing the result (the machine saying "yes") under $H_0$. Assuming that the machine correctly detects the presence of absence of neutrinos, then if the machine says "yes" under $H_0$ then it is because the machine is lying to us as a result of rolling two sixes. Thus the p-value is 1/36, so following normal quasi-Fisher scientific practice, a frequentist would reject the null hypothesis, at the 95% level of significance.

But rejecting the null hypothesis does not mean that you are entitled to accept the alternate hypothesis, so the frequentists conclusion is not justified by the analysis. Frequentist hypothesis tests embody the idea of falsificationism (sort of), you can't prove anything is true, only disprove. So if you want to assert $H_1$, you assume $H_0$ is true and only proceed if you can show that $H_0$ is inconsistent with the data. However that doesn't mean $H_1$ is true, just that it survives the test and continues as a viable hypothesis at least as far as the next test.

The Bayesian is also merely common sense, noting that there is nothing to lose by making the bet. I'm sure frequentist approaches, when the false-positive and false-negative costs are taken into account (Neyman-Peason?) would draw the same conclusion as being the best strategy in terms of long-run gain.

To summarise: Both the frequentist and Bayesian are being sloppy here: The frequentist for blindly following a recipe without considering the appropriate level of significance, false-positive/false-negative costs or the physics of the problem (i.e. not using his common sense). The Bayesian is being sloppy for not stating his priors explicitly, but then again using common sense the priors he is using are obviously correct (it is much more likely that the machine is lying than sun actually having exploded), the sloppiness is perhaps excusable.

• Rejecting the null hypothesis simply means that the observation would be unlikely IF H0 were true. You should not "accept" H1 on this basis as it is basically saying that H1 must be true because the observations would be unlikely if H0 were true. However the observations may also be unlikely under H1 (which the null ritual ignores) and H1 may be less likely than H0 a-priori (which the null ritual also ignores). Accepting hypotheses is a slippery slope towards interpreting a frequentist test as a Bayesian test, which commonly results in misunderstandings in less elementary cases. Commented Jan 17, 2013 at 10:29
• Just stumbled upon your comment. And I have the same question that @glassy had. I would like to object to your comment that if your hypotheses cover the entire space of events, here being {"Sun has gone nova", "Sun didn't go nova"}, I have difficulties understanding your point how can rejecting the "Sun has gone nova" not automatically lead to "Sun didn't go nova". Declaring a statement false implies its negation must be true. It would be great if you coul provide some reliable reference text where this point is clearly explained if possible. I'd be interested to find out more about it. Commented Nov 5, 2013 at 22:52
• Rejecting the null hypothesis does not automatically mean that the null hypothesis is probably false, just that it is reasonable to continue with the alternative hypothesis. This is (in part) because the frequentist hypothesis test does not take into account the prior probabilities of the hypotheses. More fundamentally, frequentist methods cannot be used to assign a probability to the truth of any specific hypothesis, so the link between "we can reject the null hypothesis" and "the null hypothesis is probably false" is an entirely subjective one, as far as I can see. Commented Nov 6, 2013 at 10:08
• This is kind of my point, the decision as to whether we accept H1 is subjective and is not a necessary consequence of the outcome of the test "rejection of H0 does usually lead to accepting H1". The problem is that the information you need to make the decision [P(H0), P(H1), P(Z|H1)] does not appear in the test. Essentially some of this information is partially included in setting the threshold, but this is generally incomplete and often left unstated and unjustified. The priors are still there in frequentist tests, equally subjective, but left implicit - the worst of both worlds! ;o) Commented Aug 18, 2014 at 14:34
• @Dikran, I guess we understood each other well and should stop abusing the comments section, but one last remark: I subjectively choose to accept H1 if I subjectively choose to reject [amoeba-reject] H0 based on my subjectively chosen $\alpha$ based on my subjective expert assessment of P(H1). Saying that "I am not compelled to accept H1 just because I can amoeba-reject H0" does not make any linguistic sense. But I do agree that "I am not compelled to accept H1 just because I can reject H0 at 5% level". My main point: being able to reject H0 at 5% level $\ne$ rejecting. Commented Aug 19, 2014 at 11:58

The greatest problem that I see is that there is no test statistic derived. $p$-value (with all the criticisms that Bayesian statisticians mount against it) for a value $t$ of a test statistic $T$ is defined as ${\rm Prob}[T \ge t| H_0]$ (assuming that the null is rejected for greater values of $T$, as would be a case with $\chi^2$ statistics, say). If you need to reach a decision of greater importance, you can increase the critical value and push the rejection region further up. Effectively, that's what multiple testing corrections like Bonferroni do, instructing you to use a much lower threshold for $p$-values. Instead, the frequentist statistician is stuck here with the tests of sizes on the grid of $0, 1/36, 2/36, \ldots$.

Of course, this "frequentist" approach is unscientific, as the result will hardly be reproducible. Once Sun goes supernova, it stays supernova, so the detector should keep saying "Yes" again and again. However, a repeated running of this machine is unlikely to yield the "Yes" result again. This is recognized in areas that want to present themselves as rigorous and try to reproduce their experimental results... which, as far as I understand, happens with probability anywhere between 5% (publishing the original paper was a pure type I error) and somewhere around 30-40% in some medical fields. Meta-analysis folks can fill you in with better numbers, this is just the buzz that comes across me from time to time through the statistics grapevine.

One other problem from the "proper" frequentist perspective is that rolling a die is the least powerful test, with power = significance level (if not lower; 2.7% power for the 5% significance level is nothing to boast about). Neyman-Pearson theory for t-tests agonizes over demonstrating that this is a UMPT, and a lot of high brow statistical theory (which I barely understand, I have to admit) is devoted to deriving the power curves and finding the conditions when a given test is the most powerful one in a given class. (Credits: @Dikran Marsupial mentioned the issue of power in one of the comments.)

I don't know if this troubles you, but the Bayesian statistician is shown here as the guy who knows no math and has a gambling problem. A proper Bayesian statistician would postulate the prior, discuss its degree of objectivity, derive the posterior, and demonstrate how much they learned from the data. None of that was done, so Bayesian process has been oversimplified just as much as the frequentist one has been.

This situation demonstrates the classical screening for cancer issue (and I am sure biostatisticians can describe it better than I could). When screening for a rare disease with an imperfect instrument, most of the positives come out to be false positives. Smart statisticians know that, and know better to follow up cheap and dirty screeners with more expensive and more accurate biopsies.

• If I understand your first paragraph correctly, you're saying that the threshold (0.05 in the comic) is set too high. If the comic had five dice instead of two, would you accept the threshold as being low enough? How do you decide the threshold anyway? Commented Nov 11, 2012 at 18:52
• I thought the Bayesian statistician simply took into account that the chances of the sun exploding are much, much smaller than the chances of the machine lying (so, not necessarily a clueless gambler).
– josh
Commented Nov 11, 2012 at 19:09
• More to the point: IF the sun goes nova, the winner of the bet will not be able to cash his 50$... Commented Nov 11, 2012 at 20:14 • I think the point here is the frequentist statistician is following a recipe without thinking of the true purpose of the analysis. The so-called "Bayesian" is not actually being a Bayesian, just someone using their common sense. There are plenty of examples of blind recipe following in science journals, which is why the cartoon is amusing. Commented Nov 12, 2012 at 12:46 • Lack of test statistic can't quite be the problem I don't think. A test statistic is just some function of the data. So the identity function, i.e. here the datum itself, would appear to work, at least in principle. Commented Nov 13, 2012 at 9:34 I agree with @GeorgeLewis that it may be premature to conclude the Frequentist approach is wrong - let's just rerun the neutrino detector several more times to collect more data. No need to mess around with priors. There's nothing wrong with this comic, and the reason has nothing to do with statistics. It's economics. If the frequentist is correct, the Earth will be tantamount to uninhabitable within 48 hours. The value of \$50 will be effectively null. The Bayesian, recognizing this, can make the bet knowing that his benefit is \$50 in the normal case, and marginally nothing in the sun-exploded case. • This does "have something to do with statistics" since Bayesian statistics explicitly models this as "minimizing a loss function" ;) Commented Sep 18, 2017 at 5:32 Now that CERN has decided that neutrinos are not faster than light - the electromagnetic radiation shock front would hit the earth before the neutrino change was noticed. This would have at the least (in the very short term) spectacular auroral effects. Thus the fact that it is dark would not prevent the skies from being lit up; the moon from shining excessively brightly (cf Larry Niven's "Inconstant Moon") and spectacular flashes as artificial satellites were vapourised and self combusted. All in all - perhaps the wrong test? (And whilst there may have been prior - there would be insufficient time for a realistic determination of posterior. • All the more reason to reject the hypothesis that the sun has exploded, then. :-) Commented Nov 13, 2012 at 4:46 • So this is what is meant at the end of the article when the authors say: "confirmatory studies are needed"? – DWin Commented Nov 30, 2013 at 21:28 • Actually, casually revisiting this the clear inference is in the title. The machine detects whether the sun has gone nova. There is no chance of error in the detection. The neutrino bit is irrelevant. Given that, then the statistics are such that the machine will reply "no", "no", "no"... with a 1/36 chance of being a false statement (yes) until a one off event which terminates the statistical process occurs - this will also have a 1/36 chance of being falsely reported (no), if the machine is queried during the 8 odd minute interval it takes to become evident on earth. Commented Dec 1, 2013 at 8:31 • It should probably be noted that the FTL neutrinos claim has been thoroughly refuted. en.wikipedia.org/wiki/Faster-than-light_neutrino_anomaly Commented Sep 21, 2020 at 13:37 The answer for your question: "does he correctly apply the frequentist methodology?" is no, he does not applied precisely the frequentist approach. The p-value for this problem is not exactly 1/36. We first must note that the involved hypotheses are H0: The Sun has not exploded, H1: The Sun has exploded. Then, p-value = P("the machine returns yes" | the Sun hasn't exploded). To compute this probability, we must note that "the machine returns yes" is equivalent to "the neutrino detector measures the Sun exploding AND tells the true result OR the neutrino detector does not measure the Sun exploding AND lies to us". Assuming that the dice throwing is independent of the neutrino detector measurement, we can compute the p-value by defining: p0 = P("the neutrino detector measures the Sun exploding" |the Sun hasn't exploded), Then, the p-value is p-value = p0 x 35/36 + (1-p0) x 1/36 = (1/36) x (1+ 34 x p0). For this problem, the p-value is a number between 1/36 and 35/36. The p-value is equal 1/36 if and only if p0=0. That is, a hidden assumption in this cartoon is that the detector machine will never measure the Sun exploding if the Sun hasn't exploded. Moreover, much more information should be inserted in the likelihood about external evidences of an anova explosion going on. All the Best. This is of course a frequentist 0.05 level test - the null hypothesis is rejected less than 5% of the time under the null hypothesis and even the power under the alternative is great. On the other hand prior information tells us that the sun going supernova at a by particular point in time is pretty unlikely, but that getting a lie by chance is more likely. Bottom line: there's not really anything wrong with the comic and it shows that testing implausible hypotheses leads to a high false discovery rate. Additionally, you probably want to take prior information into account in your assessment of offered bets - that's why a Bayesian posterior in combination with decision analysis is so popular. A simpler point that may be lost among all the verbose answers here is that the frequentist is depicted drawing his conclusion based upon a single sample. In practice you would never do this. Reaching a valid conclusion requires a statistically significant sample size (or in other words, science needs to be repeatable). So in practice the frequentist would run the machine multiple times and then come to a conclusion about the resulting data. Presumably this would entail asking the machine the same question several more times. And presumably if the machine is only wrong 1 out of every 36 times a clear pattern will emerge. And from that pattern (rather then from one single reading) the frequentist will draw a (fairly accurate, I would say) conclusion regarding whether or not the sun has exploded. • What do you mean by "statistically significant sample size"? – Momo Commented Nov 16, 2012 at 9:08 • @Momo - More than a single sample, that's for sure. It's not valid to observe an improbable outcome and then make conclusions that the improbable has happened without first repeating the observation to make sure it wasn't a fluke. If you want an exact number that represents a statistically significant sample size or an algorithm to determine an exact number, probably a statistician can provide one; but I'm not a statistician. Commented Nov 19, 2012 at 7:30 • I don't think there is a particular problem with having a sample size of 1 is the issue, the problem is that the test has no statistical power (i.e. the test will never reject the null hypothesis when it is false). However, this reveals a problem with the "null ritual" being lampooned in the article, which ignores the issue of statistical power (and what H1 actually is, or prior information relevant to the problem). Commented Nov 30, 2013 at 11:44 • @Dikran That is one of the best possible answers! The problem with the "frequentist" in the cartoon is that a particular statistical ritual has been followed without first conducting the necessary assessment of the properties of the test. (One might even extend your analysis by considering what a relevant loss function for this decision ought to be.) As such the cartoon neatly skewers all people who employ statistical procedures without understanding them or checking their assumptions. – whuber Commented Dec 6, 2014 at 18:55 I don't see any problem with the frequentist's approach. If the null hypothesis is rejected, the p-value is the probability of a type 1 error. A type 1 error is rejecting a true null hypothesis. In this case we have a p-value of 0.028. This means that among all the hypothesis tests with this p-value ever conducted, roughly 3 out of a hundred will reject a true null hypothesis. By construction, this would be one of those cases. Frequentists accept that sometimes they'll reject true null hypothesis or retain false null hypothesis (Type 2 errors), they've never claimed otherwise. Moreover, they precisely quantify the frequency of their erroneous inferences in the long run. Perhaps, a less confusing way of looking at this result is to exchange the roles of the hypotheses. Since the two hypotheses are simple, this is easy to do. If the null is that the sun went nova, then the p-value is 35/36=0.972. This means that this is no evidence against the hypothesis that the sun went nova, so we can't reject it based on this result. This seems more reasonable. If you are thinking. Why would anybody assume the sun went nova? I would ask you. Why would anybody carry out such an experiment if the very thought of the sun exploding seems ridiculous? I think this just shows that one has to assess the usefulness of an experiment beforehand. This experiment, for example, would be completely useless because it tests something we already know simply from looking up to the sky (Which I'm sure produces a p-value that is effectively zero). Designing a good experiment is a requirement to produce good science. If your experiment is poorly designed, then no matter what statistical inference tool you use, your results are unlikely to be useful. • Certainly, but the Bayesian can still infer a reasonable conclusion with the given data / experiment results. Sometimes you can't repeat an experiment or design it the way you want. Commented Jul 15, 2014 at 23:12 • That's a fair point, Bayesian inference can easily incorporate previous experience that makes it harder for extraordinary results to have statistical weight (It protects us against statistical flukes). However, this is also a useless experiment in the Bayesian framework. The prior is so strongly in favor of one conclusion that no result in this experiment can change it. If the prior is so strong. Why carry out an experiment with no chance of modifying it? When considering weak priors (Likely to be changed by data), I think Bayesian and Frequentist methods usually yield "comparable" results. Commented Aug 18, 2014 at 19:37 How to integrate "prior knowledge" about the sun stability in the frequentist methodology? Very interesting topic. Here are just some thoughts, not a perfect analysis... Using the Bayesian approach with a noninformative prior typically provides a statistical inference comparable to the frequentist one. Why does the Bayesian has a strong prior belief that the sun has not exploded ? Because he knows as everyone that the sun has never exploded since its beginning. We can see on some simple statistical models with conjugate priors that using a prior distribution is equivalent to use the posterior distribution derived from a noninfomative prior and preliminary experiments. The sentence above suggests that the Frequentist should conclude as the Bayesian by including the results of preliminary experiments in his model. And this is what the Bayesian actually does: his prior comes from his knowledge of the preliminary experiments ! Let$N$be the age of the sun in days, and$x_i$be the status of the sun (0 = exploded / 1 = not exploded) at day$i$. Assume the$x_i$are i.i.d Bernoulli variates with probability of succes$\theta$. The realizations of the$x_i$have been observed :$x_i=1$for all$i =1,\ldots,N$. In the current problem, we have$N+1$observations : the$x_i$and the result$y=\{\text{Yes}\}$of the detector. The natural question is : what is the probability that the sun has exploded, that is, what is$\Pr(x_{N+1}=0)$? This is$\theta$and estimating$\theta$from the available observations$x_1, \ldots, x_N$and$y$yields an estimate highly close to$1$because$N$is huge, and the "unexpected" value$y=\{\text{Yes}\}$has a negligible impact on the estimate of$\theta$. And the Bayesian intends to reflect this information through his prior distribution about$\theta$. From this perspective I don't see how to rephrase the question in terms of hypothesis testing. Taking$H_0 =\{\text{the sun has not exploded}\}\$ makes no sense cause it is a possible issue of the experiment in my interpretation, not a true/false hypothesis. Maybe this is the error of the Frequentist ?

• The passage "... he knows as everyone that the sun has never exploded since its beginning" brings to mind a story about a recent American holiday in which millions of turkeys (Meleagris gallopavo) are consumed. As time goes on, each day any intelligent turkey "knows as everyone" that she will be fed and cared for, right until that fateful (and wholly unexpected--to her) day in the middle of November! Similarly, our confidence in the stability of the sun ought to be low if all we had to rely on was the relatively short history of human observation of it.
– whuber
Commented Nov 30, 2013 at 19:06
• @whuber I would have preferred to send you this message privately. Is there a connection between your comment and the topic of discussion? I do not know if it's me that makes me ideas, but it's been several times that I feel that you comment on my answers mainly to say something against my answers. The exercise posed by the OP is the interpretation of a cartoon, and I feel that you criticize my answer as if I were talking about a real issue. Recently I did not appreciate and I still have not understood why you evoked a likely "intent" behind my answers. Commented Nov 30, 2013 at 19:26
• There was no criticism, implied or intended: sometimes a comment is really just ... a comment. It tried to highlight (in a way intended to be humorous) important questions hinted at but not addressed in your answer. I am sorry that you perceive this as either personal or an attack. BTW, this is a real question: it asks How to integrate "prior knowledge" ... in the frequentist methodology? This question evokes Hume's criticism of inductive inference and goes to issues in the philosophy of science as well as the very foundations of statistics. It is well worth some careful thought!
– whuber
Commented Nov 30, 2013 at 19:40
• It might be worth pointing out, too, that a substantial proportion of your reputation is due to my votes for your answers--which I offer as material evidence that there is no systematic behavior on my part against you.
– whuber
Commented Nov 30, 2013 at 19:49
• No, I have understand your comment. The French Google translation of your comment is already strange, but by combining my skills in English and the strange Google translations, I'm able to get a correct translation. I will be more relax next month, likely. Commented Nov 30, 2013 at 20:40

If the frequentist is set on using a p-value to determine whether the sun has exploded, his mistake is that he should be testing at a much lower significance level $$\alpha$$ than 0.05, since the claim that the sun has gone nova and that the frequentist is still alive despite photons from the nova already reaching Earth is so unlikely.

In fact, $$\alpha$$ should be so small that even if the machine is designed to always tell the truth, the frequentist still doesn't reject the null hypothesis $$H_0$$ that the sun hasn't gone nova.

Let $$q$$ be the probability that the frequentist hallucinates that the machine reports that the sun has gone nova. $$\alpha$$ should be less then $$q$$. $$H_0$$ should only be rejected if the probability of the observation under $$H_0$$ is less than $$\alpha$$, but this probability is bounded below by $$q$$, so $$H_0$$ cannot be rejected. ($$q$$ could also have been defined as the probability that the machine is broken and always reports that the sun has gone nova.)

In my view, a more correct frequentist analysis would be as follows: H0: The sun has exploded and the machine is telling the truth. H1: The sun has not exploded and the machine is lying.

The p value here is = P(sun exploded) . p(machine is telling the truth) = 0.97 . P(sun exploded)

The statistician can not conclude anything without knowing the nature of the second probability.

Although we know that P(sun exploded) is 0, because sun like stars do not explode into supernovae.