Why do people use p-values instead of computing probability of the model given data?

Question

Roughly speaking a p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it is). But wouldn't it be more natural to calculate the probability of the hypothesis given the observed outcome?

In more details. We have a coin. We flip it 20 times and we get 14 heads (14 out of 20 is what I call "outcome of experiment"). Now, our hypothesis is that the coin is fair (probabilities of head and tail are equal to each other). Now we calculate the p-value, that is equal to the probability to get 14 or more heads in 20 flips of coin. OK, now we have this probability (0.058) and we want to use this probability to judge our model (how is it likely that we have a fair coin).

But if we want to estimate the probability of the model, why don't we calculate the probability of the model given the experiment? Why do we calculate the probability of the experiment given the model (p-value)?

Answer 1

Computing the probability that the hypothesis is correct doesn't fit well within the frequentist definition of a probability (a long run frequency), which was adopted to avoid the supposed subjectivity of the Bayesian definition of a probability. The truth of a particular hypothesis is not a random variable, it is either true or it isn't and has no long run frequency. It is indeed more natural to be interested in the probability of the truth of the hypothesis, which is IMHO why p-values are often misinterpreted as the probability that the null hypothesis is true. Part of the difficulty is that from Bayes rule, we know that to compute the posterior probability that a hypothesis is true, you need to start with a prior probability that the hypothesis is true.

A Bayesian would compute the probability that the hypothesis is true, given the data (and his/her prior belief).

Essentially in deciding between frequentist and Bayesian approaches is a choice whether the supposed subjectivity of the Bayesian approach is more abhorrent than the fact that the frequentist approach generally does not give a direct answer to the question you actually want to ask - but there is room for both.

In the case of asking whether a coin is fair, i.e. the probability of a head is equal to the probability of a tail, we also have an example of a hypothesis that we know in the real world is almost certainly false right from the outset. The two sides of the coin are non-symmetric, so we should expect a slight asymmetry in the probabilities of heads and tails, so if the coin "passes" the test, it just means we don't have enough observations to be able to conclude what we already know to be true - that the coin is very slightly biased!

Answer 2

Nothing like answering a really old question, but here goes....

p-values are almost valid hypothesis tests. This is a slightly adapted exerpt taken from Jaynes's 2003 probability theory book (Repetitive experiments: probability and frequency). Suppose we have a null hypothesis $H_0$ that we wish to test. We have data $D$ and prior information $I$. Suppose that there is some unspecified hypothesis $H_A$ that we will test $H_0$ against. The posterior odds ratio for $H_A$ against $H_0$ is then given by:

$$\frac{P(H_A|DI)}{P(H_0|DI)}=\frac{P(H_A|I)}{P(H_0|I)}\times\frac{P(D|H_AI)}{P(D|H_0I)}$$

Now the first term on the right hand side is independent of the data, so the data can only influence the result via the second term. Now, we can always invent an alternative hypothesis $H_A$ such that $P(D|H_AI)=1$ - a "perfect fit" hypothesis. Thus we can use $\frac{1}{P(D|H_0I)}$ as a measure of how well the data could support any alternative hypothesis over the null. There is no alternative hypothesis that the data could support over $H_0$ by greater than $\frac{1}{P(D|H_0I)}$. We can also restrict the class of alternatives, and the change is that the $1$ is replaced by the maximised likelihood (including normalising constants) within that class. If $P(D|H_0I)$ starts to become too small, then we begin to doubt the null, because the number of alternatives between $H_0$ and $H_A$ grows (including some with non-negligible prior probabilities). But this is so very nearly what is done with p-values, but with one exception: we don't calculate the probability for $t(D)>t_0$ for some statistic $t(D)$ and some "bad" region of the statistic. We calculate the probability for $D$ - the information we actually have, rather than some subset of it, $t(D)$.

Another reason people use p-values is that they often amount to a "proper" hypothesis test, but may be easier to calculate. We can show this with the very simple example of testing the normal mean with known variance. We have data $D\equiv\{x_1,\dots,x_N\}$ with an assumed model $x_i\sim Normal(\mu,\sigma^2)$ (part of the prior information $I$). We want to test $H_0:\mu=\mu_0$. Then we have, after a little calculation:

$$P(D|H_0I)=(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{N\left[s^2+(\overline{x}-\mu_0)^2\right]}{2\sigma^2}\right)$$

Where $\overline{x}=\frac{1}{N}\sum_{i=1}^{N}x_i$ and $s^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\overline{x})^2$. This shows that the maximum value of $P(D|H_0I)$ will be achieved when $\mu_0=\overline{x}$. The maximised value is:

$$P(D|H_AI)=(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{Ns^2}{2\sigma^2}\right)$$

So we take the ratio of these two, and we get:

$$\frac{P(D|H_AI)}{P(D|H_0I)}=\frac{(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{Ns^2}{2\sigma^2}\right)}{(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{Ns^2+N(\overline{x}-\mu_0)^2}{2\sigma^2}\right)}=\exp\left(\frac{z^2}{2}\right)$$

Where $z=\sqrt{N}\frac{\overline{x}-\mu_0}{\sigma}$ is the "Z-statistic". Large values of $|z|$ cast doubt on the null hypothesis, relative to the hypothesis about the normal mean which is most strongly supported by the data. We can also see that $\overline{x}$ is the only part of the data that is needed, and thus is a sufficient statistic for the test.

The p-value approach to this problem is almost the same, but in reverse. We start with the sufficient statistic $\overline{x}$, and we caluclate its sampling distribution, which is easily shown to be $\overline{X}\sim Normal\left(\mu,\frac{\sigma^2}{N}\right)$ - where I have used a capital letter to distinguish the random variable $\overline{X}$ from the observed value $\overline{x}$. Now we need to find a region which casts doubt on the null hypothesis: this is easily seen to be those regions where $|\overline{X}-\mu_0|$ is large. So we can calculate the probability that $|\overline{X}-\mu_0|\geq |\overline{x}-\mu_0|$ as a measure of how far away the observed data is from the null hypothesis. As before, this is a simple calculation, and we get:

$$\text{p-value}=P(|\overline{X}-\mu_0|\geq |\overline{x}-\mu_0||H_0)$$ $$=1-P\left[-\sqrt{N}\frac{|\overline{x}-\mu_0|}{\sigma}\leq\sqrt{N}\frac{\overline{X}-\mu_0}{\sigma}\leq \sqrt{N}\frac{|\overline{x}-\mu_0|}{\sigma}|H_0\right]$$ $$=1-P(-|z|\leq Z\leq |z||H_0)=2\left[1-\Phi(|z|)\right]$$

Now, we can see that the p-value is a monotonic decreasing function of $|z|$, which means we essentially get the same answer as the "proper" hypothesis test. Rejecting when the p-value is below a certain threshold is the same thing as rejecting when the posterior odds is above a certain threshold. However, note that in doing the proper test, we had to define the class of alternatives, and we had to maximise a probability over that class. For the p-value, we have to find a statistic, and calculate its sampling distribution, and evaluate this at the observed value. In some sense choosing a statistic is equivalent to defining the alternative hypothesis that you are considering.

Although they are both easy things to do in this example, they are not always so easy in more complicated cases. In some cases it may be easier to choose the right statistic to use and calculate its sampling distribution. In others it may be easier to define the class of alternatives, and maximise over that class.

This simple example account for a large amount of p-value based testing, simply because so many hypothesis tests are of the "approximate normal" variety. It provides an approximate answer to your coin problem also (by using the normal approximation to the binomial). It also shows that p-values in this case will not lead you astray, at least in terms of testing a single hypothesis. In this case, we can say that a p-value is a measure of evidence against the null hypothesis.

However, the p-values have a less interpretable scale than the bayes factor - the link between p-value and the "amount" of evidence against the null is complex. p-values get too small too quickly - which makes them difficult to use properly. They tend overstate the support against the null provided by the data. If we interpret p-values as probabilities against the null - $0.1$ in odds form is $9$, when the actual evidence is $3.87$, and $0.05$ in odds form is $19$ when the actual evidence is $6.83$. Or to put it another way, using a p-value as a probability that the null is false here, is equivalent to setting the prior odds. So for p-value of $0.1$ the implied prior odds against the null are $2.33$ and for p-value of $0.05$ the implied prior odds against the null are $2.78$.

Answer 3

Your question is a great example of frequentist reasoning and is, actually quite natural. I've used this example in my classes to demonstrate the nature of hypothesis tests. I ask for a volunteer to predict the results of a coin flip. No matter what the result, I record a "correct" guess. We do this repeatedly until the class becomes suspicious.

Now, they have a null model in their head. They assume the coin is fair. Given that assumption of 50% correct when is everything is fair, every successive correct guess arouses more suspicion that the fair coin model is incorrect. A few correct guesses and they accept the role of chance. After 5 or 10 correct guesses, the class always begins to suspect that the chance of a fair coin is low. Thus it is with the nature of hypothesis testing under the frequentist model.

It is a clear and intuitive representation of the frequentist take on hypothesis testing. It is the probability of the observed data given that the null is true. It is actually quite natural as demonstrated by this easy experiment. We take it for granted that the model is 50-50 but as evidence mounts, I reject that model and suspect that there is something else at play.

So, if the probability of what I observe is low given the model I assume (the p-value) then I have some confidence in rejecting my assumed model. Thus, a p-value is a useful measure of evidence against my assumed model taking into account the role of chance.

A disclaimer: I took this exercise from a long forgotten article in, what I recall, was one of the ASA journals.

Answer 4

As a former academic who moved into practice, I'll take a shot. People use p-values because they are useful. You can't see it in textbooky examples of coin flips. Sure they're not really solid foundationally, but maybe that is not as necessary as we like to think when we're thinking academically. In the world of data, we're surrounded by a literally infinite number of possible things to look into next. With p-value computations all you need as an idea of what is uninteresting and a numerical heuristic for what sort of data might be interesting (well, plus a probability model for uninteresting). Then individually or collectively we can scan things pretty simple, rejecting the bulk of the uninteresting. The p-value allows us to say "If I don't put much priority on thinking about this otherwise, this data gives me no reason to change". I agree p-values can be misinterpreted and overinterpreted, but they're still an important part of statistics.

Answer 5

"Roughly speaking p-value gives a probability of the observed outcome of an experiment given the hypothesis (model)."

but it doesn't. Not even roughly - this fudges an essential distinction.

The model is not specified, as Raskolnikov points out, but let's assume you mean a binomial model (independent coin tosses, fixed unknown coin bias). The hypothesis is the claim that the relevant parameter in this model, the bias or probability of heads, is 0.5.

"Having this probability (p-value) we want to judge our hypothesis (how likely it is)"

We may indeed want to make this judgement but a p-value will not (and was not designed to) help us do so.

"But wouldn't it be more natural to calculate the probability of the hypothesis given the observed outcome?"

Perhaps it would. See all the discussion of Bayes above.

"[...] Now we calculate the p-value, that is equal to the probability to get 14 or more heads in 20 flips of coin. OK, now we have this probability (0.058) and we want to use this probability to judge our model (how is it likely that we have a fair coin)."

'of our hypothesis, assuming our model to be true', but essentially: yes. Large p-values indicate that the coin's behaviour is consistent with the hypothesis that it is fair. (They are also typically consistent with the hypothesis being false but so close to being true we do not have enough data to tell; see 'statistical power'.)

"But if we want to estimate the probability of the model, why we do not calculate the probability of the model given the experiment? Why do we calculate the probability of the experiment given the model (p-value)?"

We actually don't calculate the probability of the experimental results given the hypothesis in this setup. After all, the probability is only about 0.176 of seeing exactly 10 heads when the hypothesis is true, and that's the most probable value. This isn't a quantity of interest at all.

It is also relevant that we don't usually estimate the probability of the model either. Both frequentist and Bayesian answers typically assume the model is true and make their inferences about its parameters. Indeed, not all Bayesians would even in principle be interested in the probability of the model, that is: the probability that the whole situation was well modelled by a binomial distribution. They might do a lot of model checking, but never actually ask how likely the binomial was in the space of other possible models. Bayesians who care about Bayes Factors are interested, others not so much.

Answer 6

A side note to the other excellent answers: on occasion there are times we don't. For example, up until very recently, they were outright banned at the journal Epidemiology - now they are merely "strongly discouraged" and the editorial board devoted a tremendous amount of space to a discussion of them here: http://journals.lww.com/epidem/pages/collectiondetails.aspx?TopicalCollectionId=4

Answer 7

I will only add a few remarks; I agree with you that the overuse of $p$-values is harmful.

• Some people in applied stats misinterpret $p$-values, notably understanding them as the probability that the null hypotheses is true; cf these papers: P Values are not Error Probabilities and Why We Don’t Really Know What "Statistical Significance" Means: A Major Educational Failure.

• An other common misconception is that $p$-values reflect the size of effect detected, or their potential for classification, when they reflect both the size of sample and the size of effects. This leads some people to write papers to explain why variables that have been shown "strongly associated" to a character (ie with very small p values) are poor classifiers, like this one...

• To conclude, my opinion is that $p$-values are so widely used because of publications standards. In applied areas (biostats...) their size are sometimes the sole concern of some reviewers.

Answer 8

Define probability. I mean it. Before we progress any further, we need to settle on terms.

An intuitive definition of probability is a measure of uncertainty. We are uncertain whether the next coin toss will come up heads or tails. That is uncertainty in the data $D$. We are also uncertain whether the coin is fair or not. That is uncertainty about the model $M$... or you can call uncertainty about the state of the world.

To arrive at the conditional distribution $P(M|D)$, you need to have the joint distribution $P(M,D)$ -- i.e., the knowledge of the whole population of coins in circulation, how many of them are forged, and how forged coins behave (which may depend on the way the coins are spun and caught in the air).

In the particular example of coins, this is at least conceptually possible -- the government figures are available on the coins that are supposed to be fair (28$\cdot$10⁹ per year), or at least those with stable characteristics. As far as forged coins go, the scale of production of less than a million is probably not worth talking about, so $10^6/28\cdot10^9$ may be a probability that the coin you got from a cashier's register is unfair. Then you need to come up with a model of how the unfair coin works... and obtain the joint distribution, and condition on the data.

In the practical world problems with say medical conditions and the way they work, you may not be able to come up with none of these components of the joint distribution, and can't condition.

Bayesian modeling provides a way to simplify the models and come up with these joints $P(M,D)$. But the devil is in the details. If you say that the fair coin is the one with $p=0.5$, and then go ahead and specify a traditional Beta prior, and get the Beta conjugate posterior, then... surprise, surprise! $P(p=0.5)=0$ for either of these continuous distributions, no matter if your prior is $B(0.5,0.5)$ or $B(1000,1000)$. So you'd have to incorporate a point mass at $0.5$, give it a prior mass ($28\cdot10^9/(28\cdot10^9 + 10^6)$, say), and see if your data moves the posterior away from that point mass. This is a more complicated calculation that involve Metropolis-Hastings sampling rather than the more traditional Gibbs sampling.

Besides the difficulties in talking about what exactly the right models are, Bayesian methods have limited ways of dealing with model misspecification. If you don't like Gaussian errors, or you don't believe in independence of coin tosses (your hand gets tired after the first 10,000 or so tosses, so you don't toss it as high as the first 1,000 or so times, whch may affect the probabilities), all that you can do in Bayesian world is to build a more complicated model -- stick breaking priors for normal mixtures, splines in probabilities over time, whatever. But there is no direct analogue to Huber sandwich standard errors that explicitly acknowledge that the model may be misspecified, and are prepared to account for that.

Going back to my first paragraph -- again, define probability. The formal definition is the trio $<\Omega,{\mathcal F},P>$. $\Omega$ is the space of possible outcomes (combinations of models and data). $\mathcal F$ is the $\sigma$-algebra of what can be measured on that space. $P$ is the probability measure / density attached to subsets $A\subset \Omega$, $A\in\mathcal F$ -- which have to be measureable for the mathematics of probability to work. In finite dimensions, most reasonable sets are measurable -- see Borel sets, I am not going to bore you with details. With the more interesting infinite spaces (those of curves and trajectories, for instance), things get hairy very quickly. If you have a random process $X_t, t\in[0,1]$ on a unit interval in time, then the set $\{ X_t > 0, t\in[0,0.5]\}$ is not measurable, despite its apparent simplicity. (Sets like $\{ X_t > 0, t\in\{t_1, t_2, \ldots, t_k\}\}$ are measurable for finite $k$, and in fact generate the required $\sigma$-algebra. But that is not enough, apparently.) So probabilities in large dimensions may get tricky even at the level of definitions, let alone computations.

Answer 9

But if we want to estimate the probability of the model, why don't we calculate the probability of the model given the experiment?

Because we don't know how. There's infinite number of model possible, and their probability space is not defined.

Here's a practical example. Let's say I want to forecast US GDP. I get the time series, and fit a model. What is the probability that this model is true?

So, let's actually fit a random walk model into GDP series: $$\Delta\ln y_t=\mu+e_t$$ where $\mu$ is the growth rate and $e_t$ is a random error. My code below does just that, and it also produces the forecast (red) and compares it historical data (blue).

However, who said that GDP is a random walk process? What is it was a trend process? So, let's fit the trend: $$\ln y_t = c t+ e_t$$ where $c$ is the slope of the time trend. The forecast using a trend model is shown on the same chart (yellow).

Now, how would you calculate the probability that my random walk model is true? Within MLE we could calculate the likelihood of the drift $\mu$ given the data set, but that's not the probability. Second, and more importantly, how would you calculate the probability that the model is random walk with this drift knowing that it could also be a trend model? It could be any other number of models that produce this kind of dynamic.

Answer 10

IMHO, confidence intervals are a better method of expressing results. This is especially true when comparing results to be included in meta analysis and for "not significant" answers. This avoids the all too common misrepresentation of not significant results as significantly insignificant. I don't know in which "camp" that puts me, frequentist or Bayesian, and frankly don't care. What I am saying is that it makes a rather important distinction just how wide a 95% confidence interval is, for example, when comparing whether the 115 mm Hg mean blood pressure of a test series is different from a (control) mean of 120 mm Hg, it makes a world of difference if the 95% confidence interval for the difference is $\pm$ 100 mm Hg or $\pm$ 10 mm Hg, because when $\pm$ 100 mm Hg we haven't excluded anything practical; a blood pressure of 20 mm Hg is only achievable several minutes after the heart has stopped, and a blood pressure of 220 mm Hg is also problematic. Only in the latter case, does our result, 110 to 130 mm Hg as a CI, exclude anything pragmatic.

One might be taken aback to realize just how many published results from averaging three wildly different values do not seem to be different from some assumed value when the confidence in saying so admits the entire range of possible values, and a not significant $p$-value does not tell that story, such that the discussion centering on interpretation of H0 seems non-contributory to me. What are your thoughts?