Interpretation of p-value near alpha level

Question

If the alpha is set to 0.05 I have encountered many scientific publications saying that a "tendency for an effect" is present based on a p-value of 0.05 < p-value < 0.1. On the other hand, I have had statisticians who criticize me for doing so, because there is only "reject" or "not reject". Therefore, it does not make sense to distinguish between a p-value of 0.08 or 0.97. Also, some statisticians have criticized me for reporting p-values as p<0.05 because it is not precise.

My question is: how to deal with a p-values that are not below but close to my alpha?

Answer 1

There are two different approaches to interpreting statistical significance - the Fisher way, and the Neyman-Pearson way. We smush these together (into what Gerd Gigerenzer has called a 'bastardised approach'). The reason that statistical significance testing [Edit, n italics] as it is often taught and discussed doesn't seem to make sense is that, essentially, it doesn't.

Neyman-Pearson said that you pick a cutoff and you use it. It's less than the cutoff(say, 0.05) or it's not. There's no other information to convey.In NP, 0.08 and 0.97 are the same.

Fisher said you take the p-value and you treat it as the level of evidence that there is an effect. <0.2 is some evidence, but it's pretty weak; <0.1 is a bit better, but still kind of weak. <0.05 is what Fisher said is often good enough (but he also wrote that one should change one's significance level according to the situation, which no one does).

Either report the exact significance level and interpret that appropriately. Or use 0.05. Don't do this nonsense of 0.10>p>0.05.

Your p-value presents some evidence. It's not great evidence, but it's not no evidence. You shouldn't be trying to say "Yes" or "No" when maybe is an answer.

In addition, people often say that their p-value of 0.06 is "approaching statistical significance". No one says it is "Running away from significance" or that their p-value of 0.04 is "Approaching non-significance."

Answer 2

I will also add to Jeremy Miles' answer that makes a number of valid points. (I wrote initially that "I disagree with the claim that the reason that statistical significance testing doesn't seem to make sense is that, essentially, it doesn't", but Jeremy has made this more precise in the meantime.)

The p-value has a well defined mathematical meaning, which is the probability that given the null hypothesis is true, the test statistics is as far or farther away than what was observed from what is expected under the null hypothesis.

Now in language we are bound to be categorical, so we lack the words for saying how a p-value of 0.064 is different from 0.059, however we can talk for example about strong, weak, no evidence against the $H_0$ meaning here that something has happened that under the $H_0$ would happen very rarely/rarely/be very common.

If we want to make decisions (like going on to work as if the $H_0$ were true, or not true, if these two possibilities were the only ones to consider, i.e., we decide between only two courses of action) of course we need to decide how small is too small to stick with the $H_0$.

Now there are conventions like the ubiquitous $\alpha=0.05$. This is arbitrary to some extent, but you need to realise that specifying such a cutoff is necessary when making a binary decision - otherwise not (this by the way holds as well for any alternative to significance testing).

The interpretation in language of a p-value doesn't have a mathematical basis and will always be to some extent arbitrary, but once more, language is categorical, so if we use a finite number of different wordings, we are more transparent when having well defined cutoff values for them. With all the problems with which cutoff values come - for example if you put the cutoff between "very weak evidence" and "no evidence" at 0.1, you will in language distinguish between 0.99 and 0.101, but not between 0.101 and 0.103. That may not look particularly appropriate but somehow lies in the nature of the problem.

Overall you have some freedom as at least mathematics doesn't determine how you should say things, however you act more professionally when sticking to at least fairly generally accepted and transparent standards.

So using "reject/not reject" with a cutoff at 0.05 (or 0.01 in some fields) is a strong reduction of information and as such problematic, however people cannot accuse you of bending the evidence as you stick to a well established rule. That's something. It is important here though that you only really need to use this kind of binary distinction if a binary action decision is to be made. (Deciding to "believe" the $H_0$ or the alternative is in my view not a direct action, and models should not ever be "believed" in my view anyway.) Still then for those who like precision it can only be good to state the precise p-value.

It is also generally accepted to use "evidence language", although borderlines are not handled consistently all over the place. I don't think you can go very wrong by using 0.01/0.05/0.1 as cutoffs for saying there's "strong"/"some"/"weak"/"no" evidence against the $H_0$, but I have seen others. Also let's always keep in mind that categorisation is to some extent arbitrary, but also a necessity when people communicate in language.

It is also OK to differentiate even more by saying something like "There's some evidence as $p<0.05$ but it's really rather weak as in fact $p=0.046$." Or even "at $p=0.07$ we only have weak evidence against the $H_0$ but some doubt about it is justified and the effect estimator (...) is in fact quite large" (in case it is, in a subject matter sense).

Some wordings are clearly misleading such as "tends toward significance" (as (a) there's no "tending" and (b) the writer reveals implicitly that they would've wanted significance and are willing to sacrifice objectivity to make a certain impression) or "accepting" the $H_0$ (as accepting is routinely misunderstood as thinking it is true, which no p-value can ever tell you, and "all models are wrong" anyway).

PS: "Accept" can be appropriate wording in quality control applications where a batch of products may be "accepted" if a sample does not provide evidence against certain quality standards ("acceptance sampling"). The $\alpha$ in such (and some other) applications is best chosen taking into account considerations regarding costs of consequences, rather than using widespread defaults. Note also that the major benefit of the Neyman-Pearson setup is to characterise tests by error probabilities and to enable optimality theory (finding tests that have optimal power given the level). This does not mean that such tests in practice have to be interpreted exclusively in a binary reject/not reject manner; it does not "overwrite" the more precise information in the p-value.

Answer 3

I would add to the excellent answer of Jeremy Miles by saying that how you treat your p-values also strongly depends on what you want to do with them. They get a bad reputation (and rightfully so), for being the deciding factor between "Your work is worth publishing" and "Your work is garbage".

However, what you take from your p-value depends for instance on your evaluation on how much you want to avoid type I / II errors.

Let's assume you're running a large-scale clinical trial for very cancer medication which is very expensive and heavy on side effects, and in the end you test whether the treated group had better survivability than the control group. By the medical context, you are incentivised to reject the null hypothesis "The medication is no better than placebo" then and only then if you are very certain that the medication is beneficial. Something like "our p-value is 0.08 which isn't significant but look, there's a trend" isn't going to cut it there.

If you take another example, and you are trying to sift through a large dataset to find associations between, say, environmental factors and microbiome, and you find an anticorrelation between PM2.5 pollution and the abundance of Parabacteroides golsteinii. Then with your statistics you're usually not laying claim to any clear conclusions, but are simply saying "Hey, this could be worth investigating further", in other words, your main goal is hypothesis generation. If here you end up with $p=0.08$ (especially if you fastidiously corrected for multiple testing), it could still be worth mentioning that sure, it wasn't significant at the $\alpha = 0.05$ level, but maybe we should have a look. (And in a perfect world, someone interested in this would then design a dedicated study to see whether there is a link or not).

In other words, if we look at a p-value not as a magic number, but a measure of effect size, or the weight of evidence, it can be seen in a more nuanced way (including for example interpreting it while taking into account other existing evidence on a hypothesis).

Answer 4

One consideration that is missing from the discussion is the problem when you have more than one test. As your p-value defining significance increases there is a concomitant increase in the chance that one or more of your statistical tests will have a p-value below that cut-off by chance alone given that the null-hypothesis is true. While there are standard methods for controlling for this, these are seldom applied to entire manuscripts. If given a choice, I would discuss outcomes with p-values < 0.05 and give the p-value. I will not discuss the hugely awesomely gargantuanly significant p-value < 0.0000001 any different than it was significant at 0.037. I don't shift my significance level based on wanting one more significant variable to discuss. Lastly, please consider the p-value as an estimate. Visualize a 95% confidence interval about your p-value. Given the variability in your data and your sample size do you really feel that your p-value of 0.000001 is accurate to 6 decimal places? Did you report all means and standard errors to that level of accuracy?

Answer 5

Frame challenge: Focus on the data you gathered and on weighing hypotheses against each other in metrics meaningful to your domain, not on the cult of the $p$-value.

The cult of the $p$-value is a system for journal editors to avoid putting their journal's good name on too many random flukes. Set $\alpha = 0.05$, and assume everyone works in good faith with no selection bias in the garden of forking $p$aths, and only 5% of your journal will be raising false alarms about results that are actually random flukes.¹

That's all that "statistical significance" means: it's a false alarm rate under the premise of a status quo conventional wisdom null hypothesis, without reference to alternative hypotheses. Even if the d20 is perfectly fair, you get a journal article every time you roll a critical hit of 20.

Framing $\alpha = 0.05$ terms of "we require $p < 0.05$" is just a standard API for statistical tests: the same tests, computed by the same software (or computed by the same computers, back when "computer" was a job description), can be trivially scaled to different values of $\alpha$—and, perhaps, shopped around to different journals with different levels of tolerance for flukes—without any extra analysis, because under the null hypothesis, the distribution of $p$ is uniform random in $[0,1]$.

By definition, nothing about a $p$-value in general says anything about the plausibility or strength of evidence for alternative hypotheses, at least until you do a power analysis of the specific test in question to raise true alarms about specific alternative hypotheses of interest. For any statistical test of a sampling process under a null hypothesis, the $p$-value is uniformly distributed in $[0,1]$, but under alternative hypotheses, the $p$-value may have any distribution—including, for some alternatives, the same uniform distribution!

You gathered some data from a system; you have some hypotheses about the system, including, perhaps, some status quo assumptions about it in a null hypothesis. If the hypotheses make quantitative predictions, you can now compare those predictions quantitatively to weigh the hypotheses against each other—for example, if you have a likelihood function for two hypotheses, you can compute an odds ratio for the hypotheses to measure the strength of evidence for one over the other, and use that to guide a process of model selection. You might have to meet a $p$-value threshold for publication in a journal you want, but the substance of your research is about the data, methodology, and competing hypotheses.

What distinguishes metrics like odds ratios from $p$-values is that odds ratios are affected by both hypotheses you're examining, whereas $p$-values by definition are meaningful only to a null hypothesis. Now, it may happen that the computation of a $p$-value for some statistical test, $p = F(X)$, is also a useful function in the analysis of alternative hypotheses measuring the strength of evidence over the null hypothesis: you might happen to reuse the same function $F(X)$ of the data $X$ in the computation.² But that's an accident of the test in question and must be phrased in specific terms of that test to have any meaning. A $p$-value on its face confers no strength of evidence; it's only a standard interface to let editors control the fraction of random flukes in their journals.

---

_{¹ Of course, you might use the same theory for purposes other than a journal editor vetting paper submissions, but the decision theory of this rule in isolation works the same in whatever domain you apply it to.}

_{² This is what motivated Fisher's approach of publishing specific $p$-values and treating them with any kind of meaning about strength of evidence. However, although Fisher's method of combining $p$-values of several independent tests into a composite test with the same "statistical significance" (false alarm rate) is technically correct (as is the method of picking one $p$-value and discarding the rest), $p$-values as strength of evidence for alternative hypotheses are incommensurate and there is no general logic to reconcile them. Fisher attempted to create a theory of fiducial inference to address problems like this, but it is largely incoherent and has been abandoned by the wayside in the history of statistics. Unfortunately, vestiges of Fisher's abandoned theory remain in popular wisdom and textbook intellectual copypasta.}