Why do we use hypothesis tests instead of just letting people do Bayesian updates?

Question

Why do we need discretize our judgements using hypothesis tests?

Why can't we just have people report the data every time a study is done, and the p-values and effect size, and then report how the data altered their subjective probabilities?

It is more valuable for people to have their own probabilities of certain statements be true, and then update that probability when they encounter new data in a study.

Having people only do this when the data is "statistically significant" us somewhat arbitrary. Data that isn't statistically significant but still points in a certain direction is still evidence that should update your beliefs; it's just not strong enough that you can reasonably take the conclusion of the study to be true (without expecting to be wrong a lot of the time).

Some people say it is harmful, because it causes people to not publish studies that aren't statistically significant.

Everyone has their own probabilities assigned to statements, and then, when they encounter a new set of data, they update their probabilities. Maybe the data is not statistically significant, and they don't update that much, but that is better than nothing. Especially if this would lead to healthier norms in science, where all studies are published.

Answer 1

Not a terrible idea, but you need to take account of decision-necessity, analysis-time-cost, and analysis-knowledge-requirements

Hypothesis testing is useful in binary decision problems where one encounters a decision that needs to be made between two alternative courses of action --- e.g., should I take this drug or not? This is the reason we need to discretise in certain circumstances. Indeed, any time we are presented with a decision problem involving a countable number of alternative options, we need to discretise our analysis to make a decision (the binary decision problem being a special case of this).

As to the idea that you could just let people discretise their own subjective prior/posterior beliefs using Bayesian analysis, that is easy for us, but hard for others. Even for scientifically-literate users who have a sound knowledge of Bayesian analysis (and could therefore perform the required analysis), there is a substantial time cost to the process of eliciting one's own prior beliefs into probabilistic form, applying Bayes' rule to get a posterior distribution, and then discretising this distribution over the space of the relevant hypotheses. Moreover, in many circumstances it is necessary for scientifically-literate researchers to be able to communicate the evidence for a particular course of action to non-scientifically-literate users who want to make a decision. Telling the latter, "Oh, just use your subjective probabilities and apply Bayes' rule to make a decision" is not particularly helpful to them, whereas being able to summarise the evidence in a single cut-off number that the researcher has already computed for you is perhaps more helpful.$^\dagger$

Whilst I think you have missed these elements of the problem, your position is one that would have some merit if it were applied purely amongst a group of scientifically-literate users with sound knowledge of Bayesian analysis, and if this group does not mind incurring a time cost for getting results each time they read a study. The latter could be ameliorated to some degree if the researchers presenting results provide useful automated tools for eliciting priors and rapidly computing the posterior probabilities over the hypotheses, but even then there would still be an extra time cost. Amongst a group of skilled Bayesians, a methodology like the one you are presenting would probably be a reasonable way to present evidence.

---

$^\dagger$ I can already hear the objections: people are terrible at interpreting p-values so is this single value really that helpful? Well, if they can't interpret a p-value correctly, what do you think their chances are of successfully eliciting their own prior beliefs in appropriate probabilistic form, then applying Bayes rule to update, and then discretising back to a final posterior probability for the hypotheses of interest?

Answer 2

Welcome to CrossValidated.

You’re right, a “bright line” of statistical “significance” has been proven to distort interpretations and to hurt science. Peer-reviewed journals and regulatory authorities have done damage in this regard. Far better as you said to let the reader decide. Stating the p-value is better than using p < 0.05 and Bayesian probabilities are even better than that. At any point in data collection you can compute multiple Bayesian probabilities to use for decision making, e.g., Pr(A | data, prior model) where A = {effect > 0, effect < 0, effect > trivial, |effect| < epsilon}. The last assertion’s probability provides evidence for similarity.

It is wise to be more intentional and use solid designs when doing sequential testing. For example you need to have data comparability over time and no drift in how things are measured. Then you can really take advantage of Bayes with continual learning. Unlike p-values when computed multiple times, Bayesian posterior probabilities are just updates that cause previous probabilities to be obsolete, and no multiplicities are involved. This is detailed here.

An important point related to sequential analysis is that a sample size calculation is not required. Most statistical significance-based studies are based on unrealistic sample sizes whose calculations depended on unknown quantities, so the sample size was just a guesstimate.

Answer 3

Necessity is the mother of all invention. In the early 1900s there were no powerful computers which can run a Bayesian computation. Therefore, alternative methods need to be devised. Historically, Bayesian inference is older than Frequentist inference, because the Bayesian approach is far more intuitive to use. But pretty soon it became evident that the Bayesian approach is impractical unless you have a very simple problem.

With the lack of computers people had to invent clever ways to do inference while avoiding the intense computation that arises. This was the birth of the Frequentist school of statistics. It allows you to reduce your inference problems to calculation of sums, squares, and averages, along with some square roots. You compute your "test statistic" and then look up the $p$-values on an appropriate table. The tables were all pre-determined, so there is no need to recalculate them again. By reducing statistical problems down to a family of sampling-distributions it made calculations much easier to do.

Now that we have software which can run these Bayesian calculations why not revert back to the way people wanted to do statistics but where unable to? There are a number of reasons for this:

• Pedagogy. It is far easier to teach newcomers to calculate averages, squares, and look up numbers in a table/database then it is to teach them likelihood functions and Bayes theorem. The later requires more grounding in mathematics. Many researchers are uncomfortable even with logarithms.
• Old Habits. Once people learn how to something it is very hard to make them learn something new, even if the new thing is a lot better. It was hard learning statistics in the beginning, with all the tests, all the software. And now you are telling people to forget that an relearn statistics all over again? Why would they do that? So people will stick to inferior methodology.
• Research. People who have an academic profession are required to publish papers. The truth of the matter is most of those papers are trivial, not to mention most of them are probably even wrong. Once some study becomes statistically significant, or gets $p$-hacked to become significant, then it can become eligible to get published. This is the way the academic world works to a large extent. This is how people get their career. Telling people to stop doing that, publish less, and risk their careers is not something that people are willing to do. It is sad that it is like this, but that is how it works.

Answer 4

I first say that I'm not against Bayesian analyses in general. However I believe that there is too little talk about some issues with Bayesian analyses that in many situations make them less attractive than many seem to believe (which is of course no excuse to badly misuse and misinterpret NHST).

1. Although a major selling point of Bayesian analyses is that background knowledge can be incorporated appropriately through the prior, in my experience the vast majority of papers presenting Bayesian analyses don't give any or only an extremely simplistic motivation of their prior choices (I often see priors that have many ingredients, distributional shapes, hyperpriors and whatnot, and justification is restricted to one parameter choice at one level of the analysis). In many cases it is rather easy to see how existing background knowledge (for example about dependencies) is not reflected in the chosen prior; it would be much harder to incorporate it properly. Generally choosing a prior that fully reflects the available knowledge and belief is very hard if you take into account all of its features instead of taking for example for granted exchangeability, that you can use a conjugate prior, and that outliers that won't fit the chosen base model don't need to be explicitly modelled (where experience often makes it close to certain that they will occur). People shrink back from doing it not only for fear of being seen as subjective, but also because it can well open a box of Pandora if you want to do it right.

2. You often see general statements of the kind "if the sample size is large enough the prior doesn't matter", but often sample sizes are not large and/or models are complex (requiring far bigger samples). Also asymptotic theory won't normally tell you how big a sample you actually need for this to be true. Furthermore any theoretical statement of this kind requires exchangeability or a version of it, and if you don't want to assume this you are easily screwed. Proper sensitivity analyses running alternative priors that arguably are similarly compatible with the background knowledge are very rarely run (if I wanted to be mean I could suspect that they are more often run than published because the results are not as nice as people would like, but I don't really believe that it is even tried out very often; in any case, a Bayesian approach doesn't offer stronger insurance against cheating than a frequentist approach).

3. There are issues with the basic assumption of exchangeability for Bayesian analyses. Exchangeability is a simplifying assumption "for mathematical convenience" (to paraphrase de Finetti) that nobody in most real situations would believe, as it implies an a priori commitment to ignore any information in the order of observations, i.e., you force yourself before seeing the data to predict the same next outcome after a potential sequence 001011000101110010 as if you observe 000011111111000000. I know that there are Bayesian models that don't require exchangeability, but usually such models still require exchangeability at some level or of some generalised kind (such as for innovations in a time series, rather than for the observations themselves).

4. Furthermore, using proper Bayesian updating you can't get out of the model assumptions you made a priori unless you are willing to violate coherence, which is a basic requirement of subjective Bayesian probability. A coherent Bayesian can't learn that assumed exchangeability or normal (or whatever) distributional shape is inconsistent with the data. Although it is true that with frequentist model assumptions there are many very similar problems to those listed above, subjectivist Bayesian probabilities in fact do not model the real data generating process but rather subjective belief before seeing the data, so technically the data can't disprove your modelling and data will only redistribute probability in your assumed model class (and the philosophy will then state that if you properly believed in your choices a priori, you really should believe in the update whatever unexpected things can be seen in the data). The frequentist can just say "reality doesn't look like my model" and adapt it (although there are problems with this as well; in particular standard frequentist inference will be invalid if applied to the same data that caused you to change your model).

5. There are issues with "objective"/informationless priors as well. In particular they may go against existing subject matter knowledge (in which case one could argue that the resulting posterior probabilities are "wrong" because they were based on a prior that doesn't reflect the belief in the given situation properly), but also they may have undesirable implications and may not agree with each other, see Kass and Wasserman "The Selection of Prior Distributions by Formal Rules" (https://doi.org/10.2307/2291752).

6. Personally I don't think science is about "believing", and I don't think the job of probability models is to be "true" in any sense. They are models and therefore idealisations that can be used as tools, but we should keep in mind that reality is different. Now a typical subjectivist Bayesian criticism of frequentism is that because in fact reality is not like a frequentist model (with which I agree), probability should refer to subjective belief rather than the data generating processes in reality themselves. This is a flawed idea in my view because (for the reasons given above) Bayesian models of subjective belief are simplifying idealisations as well (even though often not that simple;-). I'm as much against frequentists believing their models to be "true" as I am against Bayesians forgetting that their model deviates from their "true" belief and knowledge, but my observation is that for philosophical reasons frequentists are disposed to more easily reject and replace their models than subjectivist Bayesians, because subjectivist Bayesian philosophy suggests that the data can't falsify the model. (Note that "rejecting a model" is a binary decision often using a frequentist misspecification test; something that certain Bayesians such as George Box have embraced as well.)

7. A Bayesian probability that the true value of parameter A in model B lies in set C in my view doesn't have a more straightforward interpretation than any frequentist probability statement once I doubt that model B is true and that there is any such thing as a true parameter value within it. I don't buy into the idea that Bayesian posterior probabilities "give the practitioner what they really want" as opposed to frequentist error probabilities, because these Bayesian probabilities concern parameters that don't exist in reality and are conditional on model choices that are hard to make (and that many practitioners are keen to avoid for often not so bad reasons, see above) and shouldn't be trusted regarding subjective belief.

8. If we are interested in reality and we accept that models are not true regarding both real data generating processes and subjective belief, taking the detour via modelling subjective belief, incurring the requirement to specify a prior, to say something about reality doesn't look all too attractive to me - unless there is background knowledge that really helps and that can be convincingly translated into a prior.

That said I state once more that I agree that misuse and misinterpretation of frequentist hypothesis tests are endemic, and that most of the problems listed above have analogous problems in frequentist inference. I am aware that a Bayesian analysis involving a well motivated prior can really help and do a good job in many situations. I have "accused" Bayesian models to be simplified idealisations above, but I'm not against using simplified idealisations at all, as long as we try to remain aware of what is "idealised away". I am also aware that the Bayesian can do certain things to deal with issues some listed above (not being too dogmatic and getting rid of a model that obviously doesn't fit is just one thing). I think that in both Bayesian and frequentist analysis it improves matters to be very conscious of the difference between models and reality and of problems that can be caused by this. One implication is that generally there is more uncertainty regarding our results and their interpetation than both frequentist and Bayesian inference will show us (which is still more than, say, journalists, politicians, or even certain scientists would want to communicate).

What I like about frequentist tests is that, properly interpreted as making statements about compatibility between data and model, they do not require any belief in the model. (Of course this is ignored by many who use them.)

By the way, shamelessly promoting my own work on understanding probability models without assuming them to be "true": C. Hennig: Probability Models in Statistical Data Analysis: Uses, Interpretations, Frequentism-As-Model

Answer 5

Just to address the question in the title "Why do we use hypothesis tests instead of just letting people do bayesian updates?" directly...

The reason is that Baysian probability was seen as having a subjective basis that was undesirable a-priori and scientists/statisticians wanted to put things on a more objective basis. There are two problems with this (i) Bayesianism isn't necessarily subjective (c.f. Jaynes) and (ii) frequentistm does not eliminate the subjectivity or the need for prior knowledge - ISTR that Fisher wrote that the significance level for an NHST should depend on the nature of the problem. So often the subjective elements of an NHST are simply "swept under the carpet".

There is also the point that the Bayesian method often involves bespoke integration problems, which is beyond many user's mathematical skills, and frequentist procedures are often easily implemented (without a sound understanding of the underlying framework), which is also one of the reasons why pseudo-Fisherian NHSTs proliferate.

So a lot of it is historical context.

Answer 6

I don't think this is an issue of which method/algorithm/test we use, I think this is a much more fundamental issue of the people that we are using. Every approach has advantages and disadvantages, every approach has ways to abuse it absolutely. If the people doing the analyses are knowledgeable and honest about their work, it matters little which tool they use. If the people doing these analyses have little to no knowledge and/or are untruthful/dishonest then no approach is going to fix that.

Just like you wouldn't let every person off the street do your pipe work at home so probably there should be some standards about who gets to analyse (and publish) data. But if someone shows proficiency in the field then the tools they use is less less important (there are many ways to skin a cat/cook an egg/etc.).

Answer 7

Welcome to CV. Your question is an interesting one. My answer addresses the two specific questions in the body of your question:

"I'm wondering why we need to kind of "discretize" our judgements using hypothesis tests. Why can't we just, every time a study is done, have people report the data, and the p-values and effect size, and then report how the data altered their subjective probabilities?"

Regarding your first question, it is very common in applications for hypothesis testing to be used inferentially. For example, perhaps reporting statistical significance at the 5% level and simultaneously supplying the p-value. There may not be any formal decisions as such. Supplying the p-value allows readers to make their own judgements about the evidence.

Your second question proposes a mixed procedure that reports frequentist calculations (p-values, estimated effect sizes) along with requiring researchers produce and report Bayesian posterior probabilities of the hypotheses.

As background, in my view, frequentists are interested in modelling the data-generating process itself but they are usually not interested in modelling their own uncertainty about that process. That is the key divide between frequentists and Bayesians, (and that is also why many scientists are instinctively non-Bayesian).

Implementing your proposal would raise some serious practical issues:

1. How do you coerce unwilling scientists and statisticians into modelling their uncertainty using prior probabilities when that is not the form that their uncertainty takes?
2. How do you make the scientific community or even the general community value Bayesian posterior probabilities? (How should Julie view Bob’s posterior probability?)
3. Most studies are multi-authored, so I guess there would need to be a prior probability meeting prior to data collection at which they should all report their priors, or perhaps they could try to negotiate a compromise prior? What happens if they cannot reach a compromise? Presumably each author's priors and posteriors should be published. Also, should they also provide calculation software with the final report so future readers of the report can input their own priors?

In short, this does not sound like a positive approach for science. However, my comments are not meant to be criticisms (although they are likely to be unpopular). Well done for asking such a thought-provoking question.

Answer 8

I agree with some other answers in that this could be a limitation of the toolset (the human mind), but, I think, perhaps not as simple a one as some people being too dumb to understand Bayesian inference.

The causal hypotheses themselves are attractive for the human mind to reason about. Take competitive contexts with lots of sources of the data (and of the hypotheses), such as scientific articles. Any part of the new (second hand) information is likely to be biased in a way and to a degree that would be revealed only much later. If and when that happens, one might also wish to "unlearn" what they learned from an apparently flawed source.

A computerized model can perhaps "forget" a Bayesian update (even if not organized as a Bayesian model itself - it can do so by replaying what it already learned minus the update to be forgotten), but the human mind cannot do that.

Having hypotheses explicitly articulated has two benefits:

1. Explicit hypotheses allow replication attempts. That allows detection of bad data.
2. The isolated "causal principles" behind each data set make it somewhat easier to detect the types of causal discourse that were heavily predicated on the particular bit of faulty data. This helps to reduce the cognitive setback sustained from the previously believed data.

I don't doubt that Bayesian analysis can be the perfect tool for perfect data, or for uniformly imperfect data. But: If some of your input data was actually provided by your adversaries, or if a very large proportion of the data ultimately turns out to be noise, AND you are a human, then a more structured methodology could conceivably end up more successful or at least more popular in the long run.

Answer 9

I'd like to add two reasons. One is that for the very first estimate of an effect, the p-value is useful for deciding what to do next. Do I bother collecting more data or running another study? Budgets are finite, after all.

But you seem to be limiting your question to situations where prior estimates from prior studies exist to be compared with the current finding. With respect to that context, there is another reason,

As I understand it, the idea of null-hypothesis significance testing came in an era of one-off, small samples. You needed something to tell you what would happen if you were to collect more data or draw more samples, because you could not (or it was the norm to not) do those things. For example, in my field, classic studies report results from ANOVAs with 10 observations to a cell. p-values really mattered for those contexts, but don't ask me why they didn't just get larger samples or replicate their finding like we do now – I do not know the history of that.

One ironic limitation of the p-value is that it tends to get smaller as the sample size gets larger, meaning that with a large enough sample, you pretty much always (definitely always?) end up with "statistical significance". In my field, it is especially frustrating to me that journals are requiring larger samples sizes and more within-study replications, yet still will not publish your work without p-values for each and every estimate. Do they not understand how p-values work? Probably. Or it's just inertia, I truly do not know. If a paper reports 3 studies with large samples, along with 3 successful replications (in the sense of predictions about estimates failing to be falsified), but the effect sizes were all trivially greater than zero, the p-values would still be significant if the samples were large enough. If the effect sizes were reasonably large, then frankly the p-values just get in the way - they are going to be significant still, making them redundant with the effect sizes. In studies like these, they at best add nothing, and at worst allow researchers to say "statistically significant" for effects that are clearly not significant in any other meaningful sense.

Too long/didn't read: Why don't we rely on authors to do bayesian updating themselves? 1) it's useful for deciding which of two novel studies to follow up one. 2) it used to be useful for research limited to one-off, small samples (and still is, e.g., for animal testing where the priority is limiting the number animals that need to be sacrificed), and 3) excluding 1 and 2: I don't know!

P.S. As one commenter replied, not everyone is an expert and therefore may not have a prior in their head that helps them put the current finding into context. For scholarly research, I might suggest that this could be easily overcome by e.g., journals requiring authors to report the estimates from prior literature, perhaps in a table, to give the reader that context. I personally would find this an improvement over the current practice of citing prior work verbally but then, in terms of the estimates themselves, functionally deceiving the reader into thinking that the current study is the first ever test of a given hypothesis by forcing them to use the p-value to decide whether or not to believe an effect is meaningful. Personally I would rather see the previous 5 estimates myself over the p-value of the current estimate. But that's just me!