20
$\begingroup$

Following the recent ASA and other comments on p-values and not using the term "statistically significant" what is the recommendation for presenting the results of an analysis?

For example if I carried out a t-test due to the way I was taught statistics I would say something like 'p = 0.03, the result was statistically significant' (assuming I had set significance at 0.05). Would all I need to do now is just state 'p = 0.03' and possibly include a confidence interval?

$\endgroup$
  • 2
    $\begingroup$ There is a great post over at datamethods about the communication of frequentist results. $\endgroup$ – COOLSerdash Dec 24 '19 at 13:55
  • $\begingroup$ The significance is also affected by the magnitude of the finding. If you find with 80% certainty that an apple a day cures 50-70% of cancer cases it is certainly significant. $\endgroup$ – Stian Yttervik Dec 26 '19 at 10:56
  • $\begingroup$ Despite several excellent answers. I suggest that none really confronts one of several dimensions to the question: who you are writing for and what do they know. I can conceive of an entire spectrum here from people who are statisticians or have a very strong understanding of statistics through groups that should have some understanding of statistics (but in practice many members may be ill- or misinformed) to lay audiences. The first is easiest, and I can often imagine writing something like "stated formally, the P-value from the foobar test of $\theta = 0$ is 0.03". $\endgroup$ – Nick Cox Jan 31 at 11:58
28
$\begingroup$

I don't think the objection is to just the term "statistically significant" but to the abuse of the whole concept of statistical significance testing and to the misinterpretation of results that are (or are not) statistically significant.

In particular, look at these six statements:

  1. P-values can indicate how incompatible the data are with a specified statistical model.
  2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
  3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
  4. Proper inference requires full reporting and transparency.
  5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
  6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

So, they are recommending a more comprehensive way of doing and reporting analysis than simply just giving a p value, or even a p value with a CI. I think this is wise and I don't think it ought to be controversial.

Now, going from their statement to my own views, I'd say that we often shouldn't mention the p value at all. In many cases, it doesn't provide useful information. Nearly always, we know in advance that the null is not exactly true and, quite often, we know it is not even close to true.

What to do instead? I highly recommend Robert Abelson's MAGIC criteria: Magnitude, Articulation, Generality, Interestingness and Credibility. I say much more about this in my blog post: Statistics 101: The MAGIC criteria.

(My views, unlike those of the ASA, are controversial. Many people disagree with them).

| cite | improve this answer | |
$\endgroup$
  • 12
    $\begingroup$ A big +1. Although I hold a strong disagreement with your characterization and use of null hypotheses, you have clearly articulated your position and along the way you supply useful, well-supported information. $\endgroup$ – whuber Dec 24 '19 at 16:01
8
$\begingroup$

In my opinion one of more honest yet non-technical phrasing would be something like:

The obtained result is surprising/unexpected (p = 0.03) under the assumption of no mean difference between the groups.

Or, permitting the format, it could be expanded:

The obtained difference of $\Delta m$ would be quite surprising (p = 0.03) under the scenario of two normally distributed groups with equal means and a standard deviation of $\sigma$. Since our data does not deviate too much from the distributions used within the test the obtained result suggests either that the actual means of two groups are different or that a rare sampling outcome has occurred.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ +1 for the notion that a p-value inversely measures surprisingness. I would add that it is also dependent on the appropriateness of the statistical model and the sampling. $\endgroup$ – Michael Lew Dec 24 '19 at 20:30
  • $\begingroup$ I second the +1 for mentioning that a p-value is related to surprisingness. In an Information-Theoretic sense the quantity $-log_2(p)$ measures the amount of information (supplied by the test) against the tested hypothesis. This value has a nice/intuitive interpretation, too. See bullet 4 in my A. $\endgroup$ – statmerkur Dec 30 '19 at 17:03
  • $\begingroup$ A p-value is not more valid when framed as a measure of surprise than when framed as a measure of statistical significance. $\endgroup$ – perenniallydisappointed Jan 31 at 14:54
6
$\begingroup$

I agree with the answer by Peter Flom, but would like to add an additional point on the use of the term "significance" in statistical hypothesis testing. Most hypothesis tests of interest in statistics have a null hypothesis that posits a zero value for some "effect" and an alternative hypothesis that posits a non-zero (or positive, or negative) value for that "effect". Properly construed, the p-value is a measure of evidence in favour of the alternative hypothesis, relative to the null hypothesis (and under the stipulated model). It is not a measure of the magnitude of the effect that is stipulated to be non-zero under the alternative hypothesis.$^\dagger$

In view of this, my view is that the best practice for reporting results is to refer to something like "significant evidence of a non-zero effect". The important thing here is that the quantifier "significant" is appended to the "evidence", not the "effect". In my view, saying something like "there is a significant effect" is a dangerous shorthand that commits the quantifier shift fallacy --- in lay parlance, significant evidence of a non-zero effect is very different to evidence of a significant effect. Such language invites the reader to misunderstand the meaning of the p-value, and conflate statistical significance with practical significance.

This is the most common abuse of the term "significance" I see in published papers and elsewhere. It is ubiquitous to see references to a "significant effect" or "statistically significant effect", rather than the more accurate "significance evidence of a non-zero effect".


$^\dagger$ Although obviously these things are mathematically related. Broadly speaking, the larger the true effect, the more concentrated is the distribution of the p-value near zero. Notwithstanding this fact, the p-value should not generally be used as a measure of the magnitude of the effect.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

In general, I agree with the following statements in the editorial Moving to a World Beyond "p < 0.05" which is part of the special issue Statistical Inference in the 21st Century: A World Beyond p < 0.05 of The American Statistician:

What you will NOT find in this issue is one solution that majestically replaces the outsized role that statistical significance has come to play. The statistical community has not yet converged on a simple paradigm for the use of statistical inference in scientific research—and in fact it may never do so.

We summarize our recommendations in two sentences totaling seven words: Accept uncertainty. Be thoughtful, open, and modest. Remember “ATOM.”

The authors of the 43 papers of the special issue each provide (different) answers to your question. Personally, I really like the following set of suggestions that Sander Greenland gives (copy-pasted from the editorial mentioned above):

  1. Replace any statements about statistical significance of a result with the p-value from the test, and present the p-value as an equality, not an inequality. For example, if p = 0.03 then “…was statistically significant” would be replaced by “…had p = 0.03,” and “p < 0.05” would be replaced by “p = 0.03.” (An exception: If p is so small that the accuracy becomes very poor then an inequality reflecting that limit is appropriate; e.g., depending on the sample size, p-values from normal or $\chi^2$ approximations to discrete data often lack even 1-digit accuracy when p < 0.0001.) In parallel, if p = 0.25 then “…was not statistically significant” would be replaced by “…had p = 0.25,” and “p > 0.05” would be replaced by “p = 0.25.”

  2. Present p-values for more than one possibility when testing a targeted parameter. For example, if you discuss the p-value from a test of a null hypothesis, also discuss alongside this null p-value another p-value for a plausible alternative parameter possibility (ideally the one used to calculate power in the study proposal). As another example: if you do an equivalence test, present the p-values for both the lower and upper bounds of the equivalence interval (which are used for equivalence tests based on two one-sided tests).

  3. Show confidence intervals for targeted study parameters, but also supplement them with p-values for testing relevant hypotheses (e.g., the p-values for both the null and the alternative hypotheses used for the study design or proposal, as in #2). Confidence intervals only show clearly what is in or out of the interval (i.e., a 95% interval only shows clearly what has p > 0.05 or p ≤ 0.05), but more detail is often desirable for key hypotheses under contention. [...]

  4. Supplement a focal p-value p with its Shannon information transform (s-value or surprisal) $s = -log_2(p)$. This measures the amount of information supplied by the test against the tested hypothesis (or model): Rounded off, the s-value s shows the number of heads in a row one would need to see when tossing a coin to get the same amount of information against the tosses being “fair” (independent with “heads” probability of 1/2) instead of being loaded for heads. For example, if p = 0.03, this represents $–log_2(0.03) = 5$ bits of information against the hypothesis (like getting 5 heads in a trial of “fairness” with 5 coin tosses); and if p = 0.25, this represents only $–log_2(0.25) = 2$ bits of information against the hypothesis (like getting 2 heads in a trial of “fairness” with only 2 coin tosses).

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

If we know the null hypothesis is not exactly true, yet the result is not statistically significant, then that is an issue of sample size, or statistical power. Statistical significance is not really a goal, it's a necessity that one achieves with appropriate statistical power. Given the same effect size, the results of two experiments can be statistically significant or not depending on the sample size. However, I trust the statistically significant effect size more than the other because it had a bigger sample size.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

You can just state the result: "On average, Gurples were 10 cm taller than Cheebles (Difference in Height = 10 [5, 14]; mean, 95% CI, p=0.03)."

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ While there is nothing wrong in your wording, I think it goes against the spirit of the ASA statement. The point is not so much to omit the forbidden word, but to use other ways to express the strength of evidence. $\endgroup$ – Frans Rodenburg Dec 27 '19 at 12:33
  • $\begingroup$ I included the mean and 95% CI. That’s the recommended way to express the result. The p value is just included for the old school readers. $\endgroup$ – jerlich Jan 15 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.