Imagine someone who knows Bayesian statistics but has never heard of null hypothesis significance testing. How would one best explain NHST by using Bayesian analogies?
Searching through the archives I found a nice answer to a related question, which led me to the idea to describe NHST as something like a posterior predictive check, but with a "null hypothesis" pseudo-posterior that is a Dirac $\delta$ density about $H_0$. Does that make sense, or does anyone have a better comparison?
...the idea to describe NHST as something like a posterior predictive check, but with a "null hypothesis" pseudo-posterior that is a Dirac $\delta$ density about $H_0$. Does that make sense...?
Yes. I'm not sure whether to call NHST a prior or posterior predictive check, but it is fair to see it as a form of model check. That said, a Bayesian PPC is often used to check "Is my model large enough yet, or do I need to add more nuance?" By contrast, you could says classical NHST is typically used to check "Is my model small enough yet to match the capacity of my sample size / study design, or should I simplify it further because (especially without believing in an informative prior) I just don't have enough data to estimate some parameters with adequate precision?"
Ultimately, the usual scientific reason for running a NHST is to answer the question "Is my sample size large enough to rule out sampling variation as being a major concern?" We deliberately set up a too-simple straw-man model, under which the effect we hope to learn about (say, a difference in means between treatment and control groups) isn't a true effect in the population, but could show up as an apparent effect in finite samples: $\mu_1=\mu_2$, but $\bar{x}_1\neq\bar{x}_2$.
If our sample is small, this "PPC" might lead us to conclude: "Even though we don't believe this straw-man model, the data aren't inconsistent with it. Let's design our next experiment to collect more data, so that we can rule out sampling variation as a reason to disbelieve our results."
But if our sample is large enough, we should see that the too-simple model $\mu_1=\mu_2$ typically leads to datasets that don't look like our actual sample. Then we can say, "OK, sampling variation isn't a major concern here. Now we can focus on all the other concerns: Was there random assignment? Are the measurements valid for the construct we are trying to study? etc."
Perhaps it's worth framing this a 2nd way too:
From the Bayesian point of view, a point prior often makes no sense. If your prior puts all its weight on a single $\theta$ value, the posterior will be the same, so there's no point in collecting data at all. In this sense, classical NHST is not the same as using data to update your prior probabilities for $H_0$ and $H_A$ into posterior probabilities, because it starts with a point prior solely on $H_0$. But since Bayesian methods are largely meant for updating priors into posteriors, NHST seems like nonsense to many Bayesians.
However, if you're a Bayesian who is willing to run a PPC, you are willing to admit your prior might be wrong. Maybe your initial prior is your first attempt at pinning down your beliefs on this topic, and you run the PPC to see if your prior beliefs lead to an adequately realistic model that generates adequately realistic data. If they do, you'll keep using your prior. If they don't, it might convince you that your initial prior was inadequate, and you'll revise your prior (again, NOT the same thing as updating from a prior to a posterior).
In that sense, the purpose of NHST is similar to a PPC attempting to find convincing evidence that a prior of "no effect" is unreasonable. You might not actually hold such a prior yourself, but some readers or reviewers might. By reporting a NHST, you hope to tell them: "If we had started with a simple prior of 'no effect,' a PPC would have told us that our prior was inadequate" (if you reject $H_0$), or "...was not inadequate" (if you fail to reject $H_0$).
In either case, NHST is not meant as an answer to the Bayesian's usual question "Which values of $\theta$ should I believe in?" NHST is about the study design, not really about $\theta$ itself. The Greenland & Poole article mentioned in the comments does a nice job of trying to frame p-values in more Bayesian ways, but I don't know how useful that is, because (outside of PPCs) Bayesian methods are simply tackling a very different question than NHST is.
Null hypothesis significance testing, NHST, is expressing an observed effect in terms of a probabilistic comparison with the hypothesis of an absence of the effect. An observation is statistically significant if there is a clear distinction in the support of the data for absence versus presence of an effect.
An example of performing NHST with Bayesian techniques can be the following:
Imagine there is a lady that claims that she can taste whether the tea or the milk was first added to the cup. We would like to test that claim by having her perform a blind taste test.
We test the ability of the lady, by presenting her 100 cups of tea where she has to guess whether it was tea or milk first and we record the number of correct guesses. Let's for simplicity assume that the probability of a correct guess is symmetric (independent from whether the cup was tea first or milk first).
Say that based on prior information we know that there is a probability of 0.99 to have a person that can't taste anything (the null hypothesis), and a probability of 0.01 probability that a person can taste something and the ability of this person will follow a uniform distribution.
$$\begin{array}{RL} H_0:&p=0.5\\ H_a:&p\sim U(0.5,1) \end{array}$$
We have the following likelihoods as function of the number of heads:
$$\begin{array}{rcl} \mathcal{L}(H_0,k) &=& {n \choose k} 0.5^n\\ \mathcal{L}(H_a,k) &=& \int_{0.5}^1 2 {n \choose k} p^{k}(1-p)^{n-k} dp \end{array}$$
and a likelihood ratio (where we can compute the denominator with as the incomplete beta function)
$$ \Lambda = \frac{\mathcal{L}(H_0,k)}{ \mathcal{L}(H_a,k)} = \frac{1}{2^{n+1}\int_{0.5}^1 p^{k}(1-p)^{n-k} dp}$$ and posterior odds as function of $k$ $$ \frac{P(H_0;k)}{P(H_a;k)} = \frac{P(H_0)}{P(H_a)} \frac{\mathcal{L}(H_0,k)}{ \mathcal{L}(H_a,k)} $$ will look like
If for example the lady has two thirds (67) of the tea cups guessed correctly, then this indicates an effect that she can have more than half guessed correctly. But it is not significant. The null hypothesis is just as likely as the alternative hypothesis (odds ratio around one or even slightly above it).
The classical null hypothesis significance testing is not using these priors and are using instead probability statements based on a fiducial distribution or p-value.
Those statements are independent from a prior distribution (but not on prior information, e.g. assumptions about the model describing the likelihood function), and they only regard the likelihood of the null hypothesis and aim to make this a small value in order to declare a test statistically significant.
In a way the NHST has an implicit Bayesian reasoning and assumes that data that does not support the null hypothesis is supporting instead some alternative, but unknown, alternative hypothesis. Neyman and Pearson make this more explicit by defining the fiducial distribution or p-values (which can be computed in different ways) based on a specfic alternative hypothesis.
Possibly a more simple way to regard statistical significance, and how I interpret Fisher's approach to it, is that the fiducial distribution has a probability density concentrated in a small region (and in a Bayesian analysis one could use the posterior distribution in place of the fiducial distribution).
An effect is statistically significant if the highest density region (or some other region) of a certain large amount, say 95%, does not include the parameter value relating to a zero/null effect.
Expressions of statistical significance are useful when people make point estimates. A point estimate could for instance be the maximum of the posterior distribution. But such point estimate alone does not give an indication of the entire posterior and of the difference of the estimate with other hypotheses. If we give a point estimate along with a region, then we can have a better idea about the information that the data contains about a particular parameter/hypothesis.
