I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I apologise for the wall of text, but I felt it was necessary to provide an overview of my current understanding of these concepts, before I moved on to my actual questions.

From what I've gathered, a $p$-value is simply a measure of surprise, the probability of obtaining a result at least as extreme, given that the null hypothesis is true. Fisher originally intended for it to be a continuous measure.

In the Neyman-Pearson framework, you select a significance level in advance and use this as an (arbitrary) cut-off point. The significance level is equal to the type 1 error rate. It is defined by the long run frequency, i.e. if you were to repeat an experiment 1000 times and the null hypothesis is true, about 50 of those experiments would result in a significant effect, due to the sampling variability. By choosing a significance level, we are guarding ourselves against these false positives with a certain probability. $P$-values traditionally do not appear in this framework.

If we find a $p$-value of 0.01 this does not mean that the type 1 error rate is 0.01, the type 1 error is stated a priori. I believe this is one of the major arguments in the Fisher vs N-P debate, because $p$-values are often reported as 0.05*, 0.01**, 0.001***. This could mislead people into saying that the effect is significant at a certain $p$-value, instead of at a certain significance value.

I also realise that the $p$-value is a function of the sample size. Therefore, it cannot be used as an absolute measurement. A small $p$-value could point to a small, non-relevant effect in a large sample experiment. To counter this, it is important to perform an power/effect size calculation when determining the sample size for your experiment. $P$-values tell us whether there is an effect, not how large it is. See Sullivan 2012.

My question: How can I reconcile the facts that the $p$-value is a measure of surprise (smaller = more convincing) while at the same time it cannot be viewed as an absolute measurement?

What I am confused about, is the following: can we be more confident in a small $p$-value than a large one? In the Fisherian sense, I would say yes, we are more surprised. In the N-P framework, choosing a smaller significance level would imply we are guarding ourselves more strongly against false positives.

But on the other hand, $p$-values are dependent on sample size. They are not an absolute measure. Thus we cannot simply say 0.001593 is more significant than 0.0439. Yet this what would be implied in Fisher's framework: we would be more surprised to such an extreme value. There's even discussion about the term highly significant being a misnomer: Is it wrong to refer to results as being "highly significant"?

I've heard that $p$-values in some fields of science are only considered important when they are smaller than 0.0001, whereas in other fields values around 0.01 are already considered highly significant.

Related questions:

  • $\begingroup$ Also, do not forget that a "significant" p value does not tell you anything about your theory. This is even admitted by the most ardent defenders: Precis of Statistical significance: Rationale, validity, and utility. Siu L. Chow. BEHAVIORAL AND BRAIN SCIENCES (1998) 21, 169–239 Data is interpreted when being turned into evidence. The assumptions an interpretation is based on need to be enumerated and then, if possible, checked. What is being measured? $\endgroup$ – Livid Feb 14 '15 at 20:18
  • 2
    $\begingroup$ +1, but I would encourage you to focus the question and remove the side questions. If you are interested why some people argue that confidence intervals are better than p-values, ask a separate question (but make sure it hasn't been asked before). $\endgroup$ – amoeba Feb 16 '15 at 14:44
  • 3
    $\begingroup$ Apart from that, how is your question not a duplicate of Why are lower p-values not more evidence against the null? Have you seen that thread? Perhaps you can add it to the list in the end of your post. See also a similar question What sense does it make to compare p-values to each other?, but I am reluctant to recommend that thread, because the accepted answer there is IMHO incorrect/misleading (see discussion in the comments). $\endgroup$ – amoeba Feb 16 '15 at 14:52
  • 2
    $\begingroup$ Gelman has much of relevance to say about p-values. e.g. 1. here (Gelman and Stern, Am.Stat. 2006 pdf), 2. here on his blog, 3. his blog again and perhaps also 4. here (Gelman, 2013 published comment on another paper, pdf) $\endgroup$ – Glen_b -Reinstate Monica Feb 21 '15 at 0:21
  • 2
    $\begingroup$ Thanks for the links, @Glen_b; I know the Gelman & Stern paper well and often refer to it myself, but haven't seen this 2013 paper or its discussion before. However, I would like to caution OP about interpreting Gelman & Stern in the context of his/her question. G&S offer a nice example with two studies estimating an effect as $25\pm 10$ and $10\pm 10$; in one case $p<0.01$, in another $p>0.05$, but the difference between estimates is not significant. This is important to keep in mind, but if now, following OP, we ask if the first study is more convincing, I would certainly say yes. $\endgroup$ – amoeba Feb 21 '15 at 15:26

Are smaller $p$-values "more convincing"? Yes, of course they are.

In the Fisher framework, $p$-value is a quantification of the amount of evidence against the null hypothesis. The evidence can be more or less convincing; the smaller the $p$-value, the more convincing it is. Note that in any given experiment with fixed sample size $n$, the $p$-value is monotonically related to the effect size, as @Scortchi nicely points out in his answer (+1). So smaller $p$-values correspond to larger effect sizes; of course they are more convincing!

In the Neyman-Pearson framework, the goal is to obtain a binary decision: either the evidence is "significant" or it is not. By choosing the threshold $\alpha$, we guarantee that we will not have more than $\alpha$ false positives. Note that different people can have different $\alpha$ in mind when looking at the same data; perhaps when I read a paper from a field that I am skeptical about, I would not personally consider as "significant" results with e.g. $p=0.03$ even though the authors do call them significant. My personal $\alpha$ might be set to $0.001$ or something. Obviously the lower the reported $p$-value, the more skeptical readers it will be able to convince! Hence, again, lower $p$-values are more convincing.

The currently standard practice is to combine Fisher and Neyman-Pearson approaches: if $p<\alpha$, then the results are called "significant" and the $p$-value is [exactly or approximately] reported and used as a measure of convincingness (by marking it with stars, using expressions as "highly significant", etc.); if $p>\alpha$ , then the results are called "not significant" and that's it.

This is usually referred to as a "hybrid approach", and indeed it is hybrid. Some people argue that this hybrid is incoherent; I tend to disagree. Why would it be invalid to do two valid things at the same time?

Further reading:

| cite | improve this answer | |
  • 1
    $\begingroup$ (+1) But see Section 4.4 of Michael Lew's paper: some would rather equate the amount of evidence with the likelihood than with the p-value, which makes a difference when p-values from experiments with different sampling spaces are being compared. So they talk of "indexing" or "calibrating" the evidence/likelihood. $\endgroup$ – Scortchi - Reinstate Monica Feb 18 '15 at 16:08
  • $\begingroup$ Sorry, I meant to say, more precisely, that, in this view, the relative "evidence" (or "support") for different values a parameter may take is the ratio of their likelihood functions evaluated for the observed data. So in Lew's example, one head out of six tosses is the same evidence against the null hypothesis, regardless of whether the sampling scheme is binomial or negative binomial; yet the p-values differ - you might say that under one sampling scheme you were less likely to amass as much evidence against the null. (Of course rights to the word "evidence", as with "significant", ... $\endgroup$ – Scortchi - Reinstate Monica Feb 18 '15 at 18:15
  • $\begingroup$ ... haven't yet been firmly established.) $\endgroup$ – Scortchi - Reinstate Monica Feb 18 '15 at 18:16
  • $\begingroup$ Hmmm, thanks a lot for drawing my attention to this section; I read it before but apparently missed its importance. I must say that at the moment I am confused by it. Lew writes that the p-values should not be "adjusted" by taking stopping rules into account; but I don't see any adjustments in his formulas 5-6. What would "unadjusted" p-values be? $\endgroup$ – amoeba Feb 18 '15 at 18:45
  • 1
    $\begingroup$ @Scortchi: Hmmm. I really don't understand why one of these p-values is "adjusted" and another one not; why not vice versa? I am not at all convinced by Lew's argument here, and I don't even fully understand it. Thinking about that, I found Lew's question from 2012 about the likelihood principle and p-values, and posted an answer there. The point is that one doesn't need different stopping rules to get different p-values; one can simply consider different test statistics. Perhaps we can continue to discuss there, I would appreciate your input. $\endgroup$ – amoeba Feb 18 '15 at 23:05

I don't know what's meant by smaller p-values being "better", or by us being "more confident in" them. But regarding p-values as a measure of how surprised we should be by the data, if we believed the null hypothesis, seems reasonable enough; the p-value is a monotonic function of the test statistic you've chosen to measure discrepancy with the null hypothesis in a direction you're interested in, calibrating it with respect to its properties under a relevant procedure of sampling from a population or random assignment of experimental treatments. "Significance" has become a technical term to refer to p-values' being either above or below some specified value; thus even those with no interest in specifying significance levels & accepting or rejecting hypotheses tend to avoid phrases such as "highly significant"—mere adherence to convention.

Regarding the dependence of p-values on sample size & effect size, perhaps some confusion arises because e.g. it might seem that 474 heads out of 1000 tosses should be less surprising than 2 out of 10 to someone who thinks the coin is fair—after all the sample proportion only deviates a little from 50% in the former case—yet the p-values are about the same. But true or false don't admit of degrees; the p-value's doing what's asked of it: often confidence intervals for a parameter are really what's wanted to assess how precisely an effect's been measured, & the practical or theoretical importance of its estimated magnitude.

| cite | improve this answer | |
  • 1
    $\begingroup$ +1. I think what the question was getting at, is: are smaller p-values more convincing -- that's how I understand "better" in the title (in general, the question would greatly benefit if the OP tried to focus it)? If one gets $p=0.04$ or $p=0.000004$, one would perhaps call the results "significant" in both cases, but are they more convincing in the latter case? The practice of putting "stars" near p-values assumes that they are; are they? (This is essentially asking about the often-criticized "hybrid" between Fisher and Neyman-Pearson; personally, I don't have a problem with it.) $\endgroup$ – amoeba Feb 18 '15 at 10:48

Thank you for the comments and suggested readings. I've had some more time to ponder on this problem and I believe I've managed to isolate my main sources of confusion.

  • Initially I thought there was a dichotomy between viewing the p-value as a measure of surprise versus stating that it's not an absolute measure. Now I realise these statements don't necessarily contradict each other. The former allows us to be more or less confident in the extremeness (unlikeness even?) of an observed effect, compared to other hypothetical results of the same experiment. Whereas the latter only tells us that what might be considered a convincing p-value in one experiment, might not be impressive at all in another one, e.g. if the sample sizes differ.

  • The fact that some fields of science utilise a different baseline of strong p-values, could either be a reflection of the difference in common sample sizes (astronomy, clinical, psychological experiments) and/or an attempt to convey effect size in a p-value. But the latter is an incorrect conflation of the two.

  • Significance is a yes/no question based on the alpha that was chosen prior to the experiment. A p-value can therefore not be more significant than another one, since they are either smaller or larger than the chosen significance level. On the other hand, a smaller p-value will be more convincing than a larger one (for a similar sample size/identical experiment, as mentioned in my first point).

  • Confidence intervals inherently convey the effect size, making them a nice choice to guard against the issues mentioned above.

| cite | improve this answer | |

The p-value cannot be a measure of surprise because it is only a measure of probability when the null is true. If the null is true then each possible value of p is equally likely. One cannot be surprised at any p-value prior to deciding to reject the null. Once one decides there is an effect then the p-value's meaning vanishes. One merely reports it as a link in a relatively weak inductive chain to justify the rejection, or not, of the null. But if it was rejected it actually no longer has any meaning.

| cite | improve this answer | |
  • $\begingroup$ +1 for the fact "when the null is true then every p-value is equally likely'' however, I think this holds only for continuous random variables ? $\endgroup$ – user83346 Jul 30 '16 at 7:04
  • $\begingroup$ Note that I said, every "possible" value of p is equally likely. So this is true for discreet or continuous variables. With discreet variables the number of possible values is lower. $\endgroup$ – John Jul 30 '16 at 16:07
  • $\begingroup$ are you sure that the distribution of the p-values (under $H_0$) is always uniform for discrete variables because this link seems to say someting different: stats.stackexchange.com/questions/153249/… $\endgroup$ – user83346 Jul 31 '16 at 7:38
  • $\begingroup$ I believe the leading answer demonstrates that this is a non-issue. The reason that the distribution looks non-uniform is because the possible p-values are unequally spaced. Glenn even calls it quasi-uniform. I suppose it's possible that with some very sparse tests of binomial data with small Ns then perhaps the probability of specific p-values is unequal but if you consider the probability of p-values in a given range it will be closer to uniform. $\endgroup$ – John Jul 31 '16 at 8:21
  • 1
    $\begingroup$ @amoeba: let's say that the t-test you mention tests $H_0: \mu=0.5$ and you get $p=0.0000000004$. It could be that, with the same sample you test $H_0: \mu=0.45$ and you get $p=0.0000000001$, would you then say that there is more evidence for $\mu=0.45$ ? $\endgroup$ – user83346 Jul 31 '16 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.