20
$\begingroup$

Background

Edit: I realize my use of the word "hypothesis" is confusing, I do not mean specifically a null hypothesis. I mean a proposition that something is true.

From my limited understanding, Bayesian probabilities represent beliefs. A scientist may therefore assign a belief/probability to the statement that a hypothesis is true before conducting an experiment or study, and then through formal mathematical reasoning calculate an updated belief as a numerical value (probability) when the results of the study are available.

From a frequentist point of view, a probability is not a belief. Nevertheless, it is common to find phrases along the lines of "Our study strengthens the evidence that H is true". Given that a study has produced results that gives "supporting evidence" to a hypothesis, it seems reasonable that a frequentist would have a "stronger belief" in this hypothesis. Regardless of whether the prior and posterior beliefs are represented by numbers or not (but not probabilities), it undoubtedly seems that there ought to be an order between them, such as "belief after study" > "belief before study". But exactly how this updating of beliefs would happen or how to convey how much more one believe a hypothesis is true after a study compared to before the study, is unclear to me. Granted, I am quite ignorant of statistics.


Question: Within the frequentist school of thought, is there a formal / mathematical procedure for updating beliefs?


If there is no such procedure, it seems difficult to make sense of a scientist saying that a study strengthens the evidence that something is true, beyond a "more than" and "less than" perspective. The mapping from prior and new data to beliefs seems a lot more opaque to me from the frequentist perspective compared to the Bayesian one. Sure, the Bayesian have subjective priors, but given those priors, the data and chosen analysis it seems very clear exactly how the beliefs are updated through Bayes' rule (although I know frequentists can use Bayes' rule too, just not for beliefs). On the other hand I hardly think someone employing a Bayesian methodology necessarily would actually let an obtained posterior probability represent their exact belief about something, since there can be a lot to doubt, disagree with or improve in a given analysis. I'm not trying to instill any debate between "Bayesian vs. Frequentist", I'm far too ignorant to have an opinion. Hopefully this question is not nonsensical, in that case I apologize.

$\endgroup$
3

3 Answers 3

17
$\begingroup$

If you're representing beliefs coherently with numbers you're Bayesian by definition. There are at least 46656 different kinds of Bayesian (counted here: http://fitelson.org/probability/good_bayes.pdf) but "quantitatively updating beliefs" is the one thing that unites them; if you do that, you're in the Bayesian club. Also, if you want to update beliefs, you have to update using Bayes rule; otherwise you'll be incoherent and get dutch-booked. Kinda funny how the one true path to normative rationality still admits so many varieties though.

Even though Bayesians have a monopoly on 'belief' (by definition) they don't have a monopoly on "strength of evidence". There's other ways you can quantify that, motivating the kind of language given in your example. Deborah Mayo goes into this in detail in "Statistical Inference as Severe Testing". Her preferred option is "severity". In the severity framework you don't ever quantify your beliefs, but you do get to say "this claim has been severely tested" or "this claim has not been severely tested" and you can add to severity incrementally by applying multiple tests over time. That sure feels a lot like strengthening belief; you just don't get to use that exact word to describe it (because the Bayesians own the word 'belief' now). And it really is a different thing, so it's good to avoid the possible terminology collision: what you get from high severity is good error control rates, not 'true(er) beliefs'. It behaves a lot like belief in the way it is open to continual updating though! Being picky about not calling it 'belief' is purely on the (important) technicality of not dealing in states-of-knowledge, distinguishing it from the thing Bayesians do.

Mayo writes and links to plenty more on this at https://errorstatistics.com/ Sounds like you might enjoy "Bernoulli's Fallacy" by Aubrey Clayton: it's pretty accessible popsci but really cuts to the roots of this question. Discussed in podcast form here https://www.learnbayesstats.com/episode/51-bernoullis-fallacy-crisis-modern-science-aubrey-clayton

$\endgroup$
1
  • $\begingroup$ Thanks for the links! I had actually been looking at that book by Mayo so I'll definitely try and give it a read. $\endgroup$ Apr 3 at 16:04
11
$\begingroup$

Disclaimer: I have a Bayesian bias.

The purpose of frequentist hypothesis testing is to reject the null hypothesis: that is not the same as proving the alternative hypothesis. Such experiments don't give you the "evidence that $H$ is true", even if you can hear people making claims like this. The $p$-value is the probability of observing the data $D$ more extreme than observed given that the hypothesis $H$ is true $P(D>d|H)$.

The Bayesian posterior probability is the other way around, the probability that $H$ is true given the data $P(H|D) = P(D|H) P(H) / P(D)$. So in fact in the Bayesian setting, you do update your prior belief $P(H)$ about $H$ given the observed data, while in the frequentist setting you don't. Frequentist experiments don't tell you how likely $H$ is and because of that, it doesn't give you a direct framework for updating your beliefs about it. If you rejected the hypothesis that $H=5$, it still can be anything (just not $5$), and you still don't know what it is.

There is also maximum likelihood, "find $H$ such that the likelihood of observing $D$ is highest under it", but again, it doesn't tell you what $P(H|D)$ is.

Finally, if you consider probabilities to be a measure of beliefs, you are already taking a Bayesian side. Thanks to adopting such a viewpoint, you can measure how true it is given the data. In a frequentist setting, you can only make the "assuming that it is true" claims.

$\endgroup$
26
  • 1
    $\begingroup$ There are Bayesian hypothesis tests, you can reject $H$ if there is low probability of it being true $P(H|D)$, one could argue that it even makes more sense to do it like this. $\endgroup$
    – Tim
    Apr 2 at 22:11
  • 3
    $\begingroup$ Tongue somewhat in cheek, I'd say that frequentists become subjectivist Bayesians as soon as they compute the p-value. They interpret the outcome of the test as evidence that one of the hypotheses is probably true, but to do so they have silently switched frameworks. A long run frequency is a perfectly reasonable basis for a subjectivist Bayesian belief ;o) $\endgroup$ Apr 2 at 22:18
  • 3
    $\begingroup$ IMHO it is what causes so many misinterpretations of frequentists statsitics - they are often not giving direct answers to the question we really want to ask, e.g. is this hypothesis probably true? Because we get an indirect answer, we naturally want to try and shoehorn it into being the answer to the question we wanted to ask. Frequentist procedures are fine, it is just mixing frameworks that is problematic. $\endgroup$ Apr 2 at 22:28
  • 6
    $\begingroup$ The p-value is the probability of observing the data D given that the hypothesis H is true P(D|H). I do not believe this is accurate; consider updating. Also, in a frequentist setting after the experiment you know that the data is unlikely if μ=0, but you still don't know what μ is: well, you usually have a point estimate of $\mu$, so this is what it is. Frequentists just do not have a measure of strength of belief in that $\mu$. $\endgroup$ Apr 3 at 6:18
  • 6
    $\begingroup$ @Tim your definition of p-value is subtly incorrect. The p-value is the probability of observing data more extreme than what was observed, not the probability of observing the data that were observed. One more reason to dislike p-values. Note that the probability of observing the observed data is zero in the continuous data case. $\endgroup$ Apr 3 at 14:22
1
$\begingroup$

No there is not a formal method that frequentists follow to update their beliefs. My very-abridged explanation for why this is the case is as follows, and focuses just on frequentist testing methods. We can regard hypothesis testing as addressing the question: Given the assumed probability model, is the data (statistically) consistent with the null hypothesis? To take a concrete example, suppose a claim has been made that aspirin reduces acne. A group of scientists decide to test that claim and a large well-designed experiment is undertaken. The null hypothesis is that aspirin does not reduce acne. A p-value of 0.3 is observed, and the null hypothesis is not rejected. The scientists may or may not have had views (let alone ‘beliefs’) about the null hypothesis before or after the experiment, but who cares if they did? What matters is the evidence produced by the experiment. Science progresses by testing the consensus until sufficient evidence arises to change that consensus.

$\endgroup$
9
  • $\begingroup$ "The scientists may or may not have had views (let alone ‘beliefs’) about the null hypothesis before or after the experiment, but who cares if they did? What matters is the evidence produced by the experiment." - But evidence does not speak for itself. What ultimately will decide whether aspirin is approved as a medication for treating acne are people's beliefs about its effect, not a p-value or any other statistical evidence. In my view, there needs to be a mapping from evidence to beliefs, otherwise statistical experiments are pointless. $\endgroup$ Apr 5 at 6:04
  • $\begingroup$ Maybe that falls more into the realm of the philosophy of science rather than statistics. But to me it seemed, at least superficially, that this mapping from evidence to beliefs is clearer with a Bayesian methodology. Besides, I certainly care about my own beliefs after conducting a statistical test. The fact that I personally struggle more with interpreting frequentist statistical evidence, how they should affect my beliefs, compared to Bayesian statistical evidence, was part of what prompted my question. $\endgroup$ Apr 5 at 6:08
  • $\begingroup$ Let's continue the example then. The drug regulator would not approve the use of aspirin for the treatment of acne based on that experiment alone since there was not sufficient evidence of its efficacy. (Further trials can change that decision) If a patient asks a doctor about taking aspirin for acne, the doctor can indicate that the initial claim has not been supported by research. So the test is not pointless. $\endgroup$ Apr 5 at 6:28
  • $\begingroup$ "The drug regulator would not approve the use of aspirin for the treatment of acne based on that experiment alone since there was not sufficient evidence of its efficacy" - That's a belief, as is "has not been supported by research". I do aggree with you that if I'm a scientist evaluating a study, the beliefs of the scientists conducting the study is not of much relevance. But I as a scientist will still need to form a belief based on the evidence. This formation of belief seems to be completely outside the realm of frequentist statistics, and at least partly within Bayesian statistics. $\endgroup$ Apr 5 at 7:52
  • 1
    $\begingroup$ "So the test is not pointless" - no, since people actually formed beliefs based on the test, there was a mapping from test to beliefs, but this mapping was maybe done outside the area of statistics. It still has to be done if there's going to be any point doing the test. $\endgroup$ Apr 5 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.