42
$\begingroup$

Roughly speaking a p-value gives a probability of the observed outcome of an experiment given the hypothesis (model). Having this probability (p-value) we want to judge our hypothesis (how likely it is). But wouldn't it be more natural to calculate the probability of the hypothesis given the observed outcome?

In more details. We have a coin. We flip it 20 times and we get 14 heads (14 out of 20 is what I call "outcome of experiment"). Now, our hypothesis is that the coin is fair (probabilities of head and tail are equal to each other). Now we calculate the p-value, that is equal to the probability to get 14 or more heads in 20 flips of coin. OK, now we have this probability (0.058) and we want to use this probability to judge our model (how is it likely that we have a fair coin).

But if we want to estimate the probability of the model, why don't we calculate the probability of the model given the experiment? Why do we calculate the probability of the experiment given the model (p-value)?

$\endgroup$
  • $\begingroup$ You would still have to model your experiment somehow to be able to compute the likelihood-function. $\endgroup$ – Raskolnikov Dec 17 '10 at 11:06
  • 11
    $\begingroup$ Pete Dixon wrote an article back in 1998 called "Why scientists value p-values" (psychonomic.org/backissues/1631/R382.pdf) that might be an informative read. A good follow-up would be Glover & Dixon's 2004 paper on the likelihood ratio as a replacement metric (pbr.psychonomic-journals.org/content/11/5/791.full.pdf). $\endgroup$ – Mike Lawrence Dec 17 '10 at 14:18
  • 2
    $\begingroup$ Mike, that looks suspiciously like a good answer to me. What's it doing in the comments? $\endgroup$ – Matt Parker Dec 17 '10 at 16:20
  • $\begingroup$ John D Cook posted an excellent answer to a question of mine, which i think you would find interesting: stats.stackexchange.com/questions/1164/… $\endgroup$ – doug Dec 20 '10 at 8:12
  • $\begingroup$ People don't use p-values, statisticians do. (Couldn't resist a pithy saying that's also true. Of course, once you start properly qualifying each noun, it loses its pithiness.) $\endgroup$ – Wayne Dec 30 '11 at 20:00
31
$\begingroup$

Computing the probability that the hypothesis is correct doesn't fit well within the frequentist definition of a probability (a long run frequency), which was adopted to avoid the supposed subjectivity of the Bayesian definition of a probability. The truth of a particular hypothesis is not a random variable, it is either true or it isn't and has no long run frequency. It is indeed more natural to be interested in the probability of the truth of the hypothesis, which is IMHO why p-values are often misinterpreted as the probability that the null hypothesis is true. Part of the difficulty is that from Bayes rule, we know that to compute the posterior probability that a hypothesis is true, you need to start with a prior probability that the hypothesis is true.

A Bayesian would compute the probability that the hypothesis is true, given the data (and his/her prior belief).

Essentially in deciding between frequentist and Bayesian approaches is a choice whether the supposed subjectivity of the Bayesian approach is more abhorrent than the fact that the frequentist approach generally does not give a direct answer to the question you actually want to ask - but there is room for both.

In the case of asking whether a coin is fair, i.e. the probability of a head is equal to the probability of a tail, we also have an example of a hypothesis that we know in the real world is almost certainly false right from the outset. The two sides of the coin are non-symmetric, so we should expect a slight asymmetry in the probabilities of heads and tails, so if the coin "passes" the test, it just means we don't have enough observations to be able to conclude what we already know to be true - that the coin is very slightly biased!

$\endgroup$
  • 4
    $\begingroup$ Actually, most coins are actually very close to fair, and it's hard to come up with a physically plausible way to bias them very much -- see e.g. stat.columbia.edu/~gelman/research/published/diceRev2.pdf $\endgroup$ – Ben Bolker Dec 17 '10 at 14:32
  • 8
    $\begingroup$ Being very close to fair is not the same thing as being exactly fair, which is the null hypothesis. I was pointing out one of the idiosyncrasies of hypothesis testing, namely that we often know that the null hypothesis is false, but use it anyway. A more practical test would aim to detect whether there is evidence that the coin is significantly biased, rather than significant evidence that the coin is biased. $\endgroup$ – Dikran Marsupial Dec 17 '10 at 15:13
  • 1
    $\begingroup$ Hi there, maybe I am mistaken but I thought in science, you can never say that the alternative hypothesis is true, you can only say that the null hypothesis is rejected and you accept the alternative hypothesis. To me the p value reflects the chance you will make a type 1 error, i.e. that you will reject the alternative hypothesis and accept the null hypothesis (say p=.05 or 5% of the time. It is important to distinguish between type 1 error and type 2 error, and the role that power plays in your modelling of events. $\endgroup$ – user2238 Dec 21 '10 at 7:06
  • 3
    $\begingroup$ For frequentist tests, I would use an even weaker statement, which is that you either "reject the null hypothesis" or you "fail to reject the null hypothesis", and don't accept anything. The key point being that (as in the case of the biased coin) sometimes you know a-priori that the null hypothesis is not true, you just don't have enough data to demonstrate that it isn't true; in which case it would be odd to "accept" it. Frequentist tests have type-I and type-II error rates, but that doesn't mean that they can talk of the probability of a particular hypothesis being true, as in the OP. $\endgroup$ – Dikran Marsupial Dec 22 '10 at 8:50
  • 2
    $\begingroup$ @user2238 The p-value is the chance of a type I error only when the null hypothesis is "simple" (not composite) and it happens to be true. For example, in a one-sided test of whether a coin is biased towards tails ($H_0: p\lt 0.5$), using a two-headed coin guarantees the chance of a type-I error is zero even though the p-value from any finite sample will be nonzero. $\endgroup$ – whuber Dec 30 '11 at 19:45
18
$\begingroup$

Nothing like answering a really old question, but here goes....

p-values are almost valid hypothesis tests. This is a slightly adapted exerpt taken from Jaynes's 2003 probability theory book (Repetitive experiments: probability and frequency). Suppose we have a null hypothesis $H_0$ that we wish to test. We have data $D$ and prior information $I$. Suppose that there is some unspecified hypothesis $H_A$ that we will test $H_0$ against. The posterior odds ratio for $H_A$ against $H_0$ is then given by:

$$\frac{P(H_A|DI)}{P(H_0|DI)}=\frac{P(H_A|I)}{P(H_0|I)}\times\frac{P(D|H_AI)}{P(D|H_0I)}$$

Now the first term on the right hand side is independent of the data, so the data can only influence the result via the second term. Now, we can always invent an alternative hypothesis $H_A$ such that $P(D|H_AI)=1$ - a "perfect fit" hypothesis. Thus we can use $\frac{1}{P(D|H_0I)}$ as a measure of how well the data could support any alternative hypothesis over the null. There is no alternative hypothesis that the data could support over $H_0$ by greater than $\frac{1}{P(D|H_0I)}$. We can also restrict the class of alternatives, and the change is that the $1$ is replaced by the maximised likelihood (including normalising constants) within that class. If $P(D|H_0I)$ starts to become too small, then we begin to doubt the null, because the number of alternatives between $H_0$ and $H_A$ grows (including some with non-negligible prior probabilities). But this is so very nearly what is done with p-values, but with one exception: we don't calculate the probability for $t(D)>t_0$ for some statistic $t(D)$ and some "bad" region of the statistic. We calculate the probability for $D$ - the information we actually have, rather than some subset of it, $t(D)$.

Another reason people use p-values is that they often amount to a "proper" hypothesis test, but may be easier to calculate. We can show this with the very simple example of testing the normal mean with known variance. We have data $D\equiv\{x_1,\dots,x_N\}$ with an assumed model $x_i\sim Normal(\mu,\sigma^2)$ (part of the prior information $I$). We want to test $H_0:\mu=\mu_0$. Then we have, after a little calculation:

$$P(D|H_0I)=(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{N\left[s^2+(\overline{x}-\mu_0)^2\right]}{2\sigma^2}\right)$$

Where $\overline{x}=\frac{1}{N}\sum_{i=1}^{N}x_i$ and $s^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\overline{x})^2$. This shows that the maximum value of $P(D|H_0I)$ will be achieved when $\mu_0=\overline{x}$. The maximised value is:

$$P(D|H_AI)=(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{Ns^2}{2\sigma^2}\right)$$

So we take the ratio of these two, and we get:

$$\frac{P(D|H_AI)}{P(D|H_0I)}=\frac{(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{Ns^2}{2\sigma^2}\right)}{(2\pi\sigma^2)^{-\frac{N}{2}}\exp\left(-\frac{Ns^2+N(\overline{x}-\mu_0)^2}{2\sigma^2}\right)}=\exp\left(\frac{z^2}{2}\right)$$

Where $z=\sqrt{N}\frac{\overline{x}-\mu_0}{\sigma}$ is the "Z-statistic". Large values of $|z|$ cast doubt on the null hypothesis, relative to the hypothesis about the normal mean which is most strongly supported by the data. We can also see that $\overline{x}$ is the only part of the data that is needed, and thus is a sufficient statistic for the test.

The p-value approach to this problem is almost the same, but in reverse. We start with the sufficient statistic $\overline{x}$, and we caluclate its sampling distribution, which is easily shown to be $\overline{X}\sim Normal\left(\mu,\frac{\sigma^2}{N}\right)$ - where I have used a capital letter to distinguish the random variable $\overline{X}$ from the observed value $\overline{x}$. Now we need to find a region which casts doubt on the null hypothesis: this is easily seen to be those regions where $|\overline{X}-\mu_0|$ is large. So we can calculate the probability that $|\overline{X}-\mu_0|\geq |\overline{x}-\mu_0|$ as a measure of how far away the observed data is from the null hypothesis. As before, this is a simple calculation, and we get:

$$\text{p-value}=P(|\overline{X}-\mu_0|\geq |\overline{x}-\mu_0||H_0)$$ $$=1-P\left[-\sqrt{N}\frac{|\overline{x}-\mu_0|}{\sigma}\leq\sqrt{N}\frac{\overline{X}-\mu_0}{\sigma}\leq \sqrt{N}\frac{|\overline{x}-\mu_0|}{\sigma}|H_0\right]$$ $$=1-P(-|z|\leq Z\leq |z||H_0)=2\left[1-\Phi(|z|)\right]$$

Now, we can see that the p-value is a monotonic decreasing function of $|z|$, which means we essentially get the same answer as the "proper" hypothesis test. Rejecting when the p-value is below a certain threshold is the same thing as rejecting when the posterior odds is above a certain threshold. However, note that in doing the proper test, we had to define the class of alternatives, and we had to maximise a probability over that class. For the p-value, we have to find a statistic, and calculate its sampling distribution, and evaluate this at the observed value. In some sense choosing a statistic is equivalent to defining the alternative hypothesis that you are considering.

Although they are both easy things to do in this example, they are not always so easy in more complicated cases. In some cases it may be easier to choose the right statistic to use and calculate its sampling distribution. In others it may be easier to define the class of alternatives, and maximise over that class.

This simple example account for a large amount of p-value based testing, simply because so many hypothesis tests are of the "approximate normal" variety. It provides an approximate answer to your coin problem also (by using the normal approximation to the binomial). It also shows that p-values in this case will not lead you astray, at least in terms of testing a single hypothesis. In this case, we can say that a p-value is a measure of evidence against the null hypothesis.

However, the p-values have a less interpretable scale than the bayes factor - the link between p-value and the "amount" of evidence against the null is complex. p-values get too small too quickly - which makes them difficult to use properly. They tend overstate the support against the null provided by the data. If we interpret p-values as probabilities against the null - $0.1$ in odds form is $9$, when the actual evidence is $3.87$, and $0.05$ in odds form is $19$ when the actual evidence is $6.83$. Or to put it another way, using a p-value as a probability that the null is false here, is equivalent to setting the prior odds. So for p-value of $0.1$ the implied prior odds against the null are $2.33$ and for p-value of $0.05$ the implied prior odds against the null are $2.78$.

$\endgroup$
  • 4
    $\begingroup$ +1. "...choosing a statistic is equivalent to defining the alternative hypothesis that you are considering" strikes me as a deep insight. $\endgroup$ – whuber Dec 30 '11 at 19:46
  • $\begingroup$ Good answer. It is worth noting (though obvious) that working with a class of alternatives that is larger than $k$ for some small $k$ can often be computationally prohibitive, let alone if one has to work with an infinite or uncountable number of alternatives, which may also occur in practice. A big plus of the p-value approach is that it is often (usually?) computationally simple/tractable. $\endgroup$ – Faheem Mitha Dec 31 '11 at 6:46
  • 1
    $\begingroup$ @faheemmitha- you are right about the combinatorical explosion, however this does not occur for the approach i describe (in fact you can show that the bayes approach is effectively defining residuals). This is because we only need to define the class then maximise. We do not need to evaluate each alternative, just find the best one. $\endgroup$ – probabilityislogic Jan 7 '12 at 11:53
  • $\begingroup$ Why is this answer Community Wiki? $\endgroup$ – amoeba Aug 11 '16 at 20:12
10
$\begingroup$

As a former academic who moved into practice, I'll take a shot. People use p-values because they are useful. You can't see it in textbooky examples of coin flips. Sure they're not really solid foundationally, but maybe that is not as necessary as we like to think when we're thinking academically. In the world of data, we're surrounded by a literally infinite number of possible things to look into next. With p-value computations all you need as an idea of what is uninteresting and a numerical heuristic for what sort of data might be interesting (well, plus a probability model for uninteresting). Then individually or collectively we can scan things pretty simple, rejecting the bulk of the uninteresting. The p-value allows us to say "If I don't put much priority on thinking about this otherwise, this data gives me no reason to change". I agree p-values can be misinterpreted and overinterpreted, but they're still an important part of statistics.

$\endgroup$
10
$\begingroup$

Your question is a great example of frequentist reasoning and is, actually quite natural. I've used this example in my classes to demonstrate the nature of hypothesis tests. I ask for a volunteer to predict the results of a coin flip. No matter what the result, I record a "correct" guess. We do this repeatedly until the class becomes suspicious.

Now, they have a null model in their head. They assume the coin is fair. Given that assumption of 50% correct when is everything is fair, every successive correct guess arouses more suspicion that the fair coin model is incorrect. A few correct guesses and they accept the role of chance. After 5 or 10 correct guesses, the class always begins to suspect that the chance of a fair coin is low. Thus it is with the nature of hypothesis testing under the frequentist model.

It is a clear and intuitive representation of the frequentist take on hypothesis testing. It is the probability of the observed data given that the null is true. It is actually quite natural as demonstrated by this easy experiment. We take it for granted that the model is 50-50 but as evidence mounts, I reject that model and suspect that there is something else at play.

So, if the probability of what I observe is low given the model I assume (the p-value) then I have some confidence in rejecting my assumed model. Thus, a p-value is a useful measure of evidence against my assumed model taking into account the role of chance.

A disclaimer: I took this exercise from a long forgotten article in, what I recall, was one of the ASA journals.

$\endgroup$
  • $\begingroup$ Brett, this is interesting and a great example. The model here to me seems to be that people expect the order of heads and tails to occur in a random fashion. For example, if I see 5 heads in a row, I infer that this is an example of a non-random process. In fact, and I may be wrong here, the probability of a toin coss (assuming randomness) is 50% heads and 50% tails, and this is completely independent of the previous result. The point is that if we threw a coin 50000 times, and the first 25000 were heads, provided the remaining 25000 were tails, this still reflects a lack of bias $\endgroup$ – user2238 Dec 21 '10 at 7:26
  • $\begingroup$ @user2238: Your last statement is true, but it would be extrordinarily rare. In fact, seeing a run of 5 heads in 5 tosses would happen just 3% of the time if the coin is fair. It is always possible that the null is true and we have witnessed a rare event. $\endgroup$ – Brett Oct 13 '11 at 15:43
6
$\begingroup$

"Roughly speaking p-value gives a probability of the observed outcome of an experiment given the hypothesis (model)."

but it doesn't. Not even roughly - this fudges an essential distinction.

The model is not specified, as Raskolnikov points out, but let's assume you mean a binomial model (independent coin tosses, fixed unknown coin bias). The hypothesis is the claim that the relevant parameter in this model, the bias or probability of heads, is 0.5.

"Having this probability (p-value) we want to judge our hypothesis (how likely it is)"

We may indeed want to make this judgement but a p-value will not (and was not designed to) help us do so.

"But wouldn't it be more natural to calculate the probability of the hypothesis given the observed outcome?"

Perhaps it would. See all the discussion of Bayes above.

"[...] Now we calculate the p-value, that is equal to the probability to get 14 or more heads in 20 flips of coin. OK, now we have this probability (0.058) and we want to use this probability to judge our model (how is it likely that we have a fair coin)."

'of our hypothesis, assuming our model to be true', but essentially: yes. Large p-values indicate that the coin's behaviour is consistent with the hypothesis that it is fair. (They are also typically consistent with the hypothesis being false but so close to being true we do not have enough data to tell; see 'statistical power'.)

"But if we want to estimate the probability of the model, why we do not calculate the probability of the model given the experiment? Why do we calculate the probability of the experiment given the model (p-value)?"

We actually don't calculate the probability of the experimental results given the hypothesis in this setup. After all, the probability is only about 0.176 of seeing exactly 10 heads when the hypothesis is true, and that's the most probable value. This isn't a quantity of interest at all.

It is also relevant that we don't usually estimate the probability of the model either. Both frequentist and Bayesian answers typically assume the model is true and make their inferences about its parameters. Indeed, not all Bayesians would even in principle be interested in the probability of the model, that is: the probability that the whole situation was well modelled by a binomial distribution. They might do a lot of model checking, but never actually ask how likely the binomial was in the space of other possible models. Bayesians who care about Bayes Factors are interested, others not so much.

$\endgroup$
  • 2
    $\begingroup$ Hmm, two down votes. If the answer is so bad it would be nice have some commentary. $\endgroup$ – conjugateprior Dec 19 '10 at 10:57
  • $\begingroup$ I liked this answer. Sometimes people down vote answers because it is not similar to a textbook and try to rid all sites of discussions containing a taint of common sense or laymen like description. $\endgroup$ – Vass Sep 21 '11 at 13:00
  • $\begingroup$ I didn’t downvote but I think a problem is that your point is not clear. $\endgroup$ – Elvis Dec 30 '11 at 23:04
6
$\begingroup$

A side note to the other excellent answers: on occasion there are times we don't. For example, up until very recently, they were outright banned at the journal Epidemiology - now they are merely "strongly discouraged" and the editorial board devoted a tremendous amount of space to a discussion of them here: http://journals.lww.com/epidem/pages/collectiondetails.aspx?TopicalCollectionId=4

$\endgroup$
3
$\begingroup$

I will only add a few remarks; I agree with you that the overuse of $p$-values is harmful.

  • Some people in applied stats misinterpret $p$-values, notably understanding them as the probability that the null hypotheses is true; cf these papers: P Values are not Error Probabilities and Why We Don’t Really Know What "Statistical Significance" Means: A Major Educational Failure.

  • An other common misconception is that $p$-values reflect the size of effect detected, or their potential for classification, when they reflect both the size of sample and the size of effects. This leads some people to write papers to explain why variables that have been shown "strongly associated" to a character (ie with very small p values) are poor classifiers, like this one...

  • To conclude, my opinion is that $p$-values are so widely used because of publications standards. In applied areas (biostats...) their size are sometimes the sole concern of some reviewers.

$\endgroup$
2
$\begingroup$

Define probability. I mean it. Before we progress any further, we need to settle on terms.

An intuitive definition of probability is a measure of uncertainty. We are uncertain whether the next coin toss will come up heads or tails. That is uncertainty in the data $D$. We are also uncertain whether the coin is fair or not. That is uncertainty about the model $M$... or you can call uncertainty about the state of the world.

To arrive at the conditional distribution $P(M|D)$, you need to have the joint distribution $P(M,D)$ -- i.e., the knowledge of the whole population of coins in circulation, how many of them are forged, and how forged coins behave (which may depend on the way the coins are spun and caught in the air).

In the particular example of coins, this is at least conceptually possible -- the government figures are available on the coins that are supposed to be fair (28$\cdot$109 per year), or at least those with stable characteristics. As far as forged coins go, the scale of production of less than a million is probably not worth talking about, so $10^6/28\cdot10^9$ may be a probability that the coin you got from a cashier's register is unfair. Then you need to come up with a model of how the unfair coin works... and obtain the joint distribution, and condition on the data.

In the practical world problems with say medical conditions and the way they work, you may not be able to come up with none of these components of the joint distribution, and can't condition.

Bayesian modeling provides a way to simplify the models and come up with these joints $P(M,D)$. But the devil is in the details. If you say that the fair coin is the one with $p=0.5$, and then go ahead and specify a traditional Beta prior, and get the Beta conjugate posterior, then... surprise, surprise! $P(p=0.5)=0$ for either of these continuous distributions, no matter if your prior is $B(0.5,0.5)$ or $B(1000,1000)$. So you'd have to incorporate a point mass at $0.5$, give it a prior mass ($28\cdot10^9/(28\cdot10^9 + 10^6)$, say), and see if your data moves the posterior away from that point mass. This is a more complicated calculation that involve Metropolis-Hastings sampling rather than the more traditional Gibbs sampling.

Besides the difficulties in talking about what exactly the right models are, Bayesian methods have limited ways of dealing with model misspecification. If you don't like Gaussian errors, or you don't believe in independence of coin tosses (your hand gets tired after the first 10,000 or so tosses, so you don't toss it as high as the first 1,000 or so times, whch may affect the probabilities), all that you can do in Bayesian world is to build a more complicated model -- stick breaking priors for normal mixtures, splines in probabilities over time, whatever. But there is no direct analogue to Huber sandwich standard errors that explicitly acknowledge that the model may be misspecified, and are prepared to account for that.

Going back to my first paragraph -- again, define probability. The formal definition is the trio $<\Omega,{\mathcal F},P>$. $\Omega$ is the space of possible outcomes (combinations of models and data). $\mathcal F$ is the $\sigma$-algebra of what can be measured on that space. $P$ is the probability measure / density attached to subsets $A\subset \Omega$, $A\in\mathcal F$ -- which have to be measureable for the mathematics of probability to work. In finite dimensions, most reasonable sets are measurable -- see Borel sets, I am not going to bore you with details. With the more interesting infinite spaces (those of curves and trajectories, for instance), things get hairy very quickly. If you have a random process $X_t, t\in[0,1]$ on a unit interval in time, then the set $\{ X_t > 0, t\in[0,0.5]\}$ is not measurable, despite its apparent simplicity. (Sets like $\{ X_t > 0, t\in\{t_1, t_2, \ldots, t_k\}\}$ are measurable for finite $k$, and in fact generate the required $\sigma$-algebra. But that is not enough, apparently.) So probabilities in large dimensions may get tricky even at the level of definitions, let alone computations.

$\endgroup$
1
$\begingroup$

But if we want to estimate the probability of the model, why don't we calculate the probability of the model given the experiment?

Because we don't know how. There's infinite number of model possible, and their probability space is not defined.

Here's a practical example. Let's say I want to forecast US GDP. I get the time series, and fit a model. What is the probability that this model is true?

So, let's actually fit a random walk model into GDP series: $$\Delta\ln y_t=\mu+e_t$$ where $\mu$ is the growth rate and $e_t$ is a random error. My code below does just that, and it also produces the forecast (red) and compares it historical data (blue).

enter image description here

However, who said that GDP is a random walk process? What is it was a trend process? So, let's fit the trend: $$\ln y_t = c t+ e_t$$ where $c$ is the slope of the time trend. The forecast using a trend model is shown on the same chart (yellow).

Now, how would you calculate the probability that my random walk model is true? Within MLE we could calculate the likelihood of the drift $\mu$ given the data set, but that's not the probability. Second, and more importantly, how would you calculate the probability that the model is random walk with this drift knowing that it could also be a trend model? It could be any other number of models that produce this kind of dynamic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.