Following the recent ASA and other comments on p-values and not using the term "statistically significant" what is the recommendation for presenting the results of an analysis?
For example, after doing a t-test, due to the way I was taught, I would say something like "p = 0.03, the result was statistically significant" (assuming I had set the significance level at 0.05). Would all I need to do now is just state 'p = 0.03' and possibly include a confidence interval?
I don't think the objection is to just the term "statistically significant" but to the abuse of the whole concept of statistical significance testing and to the misinterpretation of results that are (or are not) statistically significant.
In particular, look at these six statements:
- P-values can indicate how incompatible the data are with a specified statistical model.
- P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
- Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
- Proper inference requires full reporting and transparency.
- A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
- By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
So, they are recommending a more comprehensive way of doing and reporting analysis than simply just giving a p value, or even a p value with a CI. I think this is wise and I don't think it ought to be controversial.
Now, going from their statement to my own views, I'd say that we often shouldn't mention the p value at all. In many cases, it doesn't provide useful information. Nearly always, we know in advance that the null is not exactly true and, quite often, we know it is not even close to true.
What to do instead? I highly recommend Robert Abelson's MAGIC criteria: Magnitude, Articulation, Generality, Interestingness and Credibility. I say much more about this in my blog post: Statistics 101: The MAGIC criteria.
(My views, unlike those of the ASA, are controversial. Many people disagree with them).
In my opinion one of more honest yet non-technical phrasing would be something like:
The obtained result is surprising/unexpected (p = 0.03) under the assumption of no mean difference between the groups.
Or, permitting the format, it could be expanded:
The obtained difference of $\Delta m$ would be quite surprising (p = 0.03) under the scenario of two normally distributed groups with equal means and a standard deviation of $\sigma$. Since our data does not deviate too much from the distributions used within the test the obtained result suggests either that the actual means of two groups are different or that a rare sampling outcome has occurred.
I agree with the answer by Peter Flom, but would like to add an additional point on the use of the term "significance" in statistical hypothesis testing. Most hypothesis tests of interest in statistics have a null hypothesis that posits a zero value for some "effect" and an alternative hypothesis that posits a non-zero (or positive, or negative) value for that "effect". Properly construed, the p-value is a measure of evidence in favour of the alternative hypothesis, relative to the null hypothesis (and under the stipulated model). It is not a measure of the magnitude of the effect that is stipulated to be non-zero under the alternative hypothesis.$^\dagger$
In view of this, my view is that the best practice for reporting results is to refer to something like "significant evidence of a non-zero effect". The important thing here is that the quantifier "significant" is appended to the "evidence", not the "effect". In my view, saying something like "there is a significant effect" is a dangerous shorthand that commits the quantifier shift fallacy --- in lay parlance, significant evidence of a non-zero effect is very different to evidence of a significant effect. Such language invites the reader to misunderstand the meaning of the p-value, and conflate statistical significance with practical significance.
This is the most common abuse of the term "significance" I see in published papers and elsewhere. It is ubiquitous to see references to a "significant effect" or "statistically significant effect", rather than the more accurate "significance evidence of a non-zero effect".
$^\dagger$ Although obviously these things are mathematically related. Broadly speaking, the larger the true effect, the more concentrated is the distribution of the p-value near zero. Notwithstanding this fact, the p-value should not generally be used as a measure of the magnitude of the effect.
In general, I agree with the following statements in the editorial Moving to a World Beyond "p < 0.05" which is part of the special issue Statistical Inference in the 21st Century: A World Beyond p < 0.05 of The American Statistician:
What you will NOT find in this issue is one solution that majestically replaces the outsized role that statistical significance has come to play. The statistical community has not yet converged on a simple paradigm for the use of statistical inference in scientific research—and in fact it may never do so.
We summarize our recommendations in two sentences totaling seven words: Accept uncertainty. Be thoughtful, open, and modest. Remember “ATOM.”
The authors of the 43 papers of the special issue each provide (different) answers to your question. Personally, I really like the following set of suggestions that Sander Greenland gives (copy-pasted from the editorial mentioned above):
Replace any statements about statistical significance of a result with the p-value from the test, and present the p-value as an equality, not an inequality. For example, if p = 0.03 then “…was statistically significant” would be replaced by “…had p = 0.03,” and “p < 0.05” would be replaced by “p = 0.03.” (An exception: If p is so small that the accuracy becomes very poor then an inequality reflecting that limit is appropriate; e.g., depending on the sample size, p-values from normal or $\chi^2$ approximations to discrete data often lack even 1-digit accuracy when p < 0.0001.) In parallel, if p = 0.25 then “…was not statistically significant” would be replaced by “…had p = 0.25,” and “p > 0.05” would be replaced by “p = 0.25.”
Present p-values for more than one possibility when testing a targeted parameter. For example, if you discuss the p-value from a test of a null hypothesis, also discuss alongside this null p-value another p-value for a plausible alternative parameter possibility (ideally the one used to calculate power in the study proposal). As another example: if you do an equivalence test, present the p-values for both the lower and upper bounds of the equivalence interval (which are used for equivalence tests based on two one-sided tests).
Show confidence intervals for targeted study parameters, but also supplement them with p-values for testing relevant hypotheses (e.g., the p-values for both the null and the alternative hypotheses used for the study design or proposal, as in #2). Confidence intervals only show clearly what is in or out of the interval (i.e., a 95% interval only shows clearly what has p > 0.05 or p ≤ 0.05), but more detail is often desirable for key hypotheses under contention. [...]
Supplement a focal p-value p with its Shannon information transform (s-value or surprisal) $s = -log_2(p)$. This measures the amount of information supplied by the test against the tested hypothesis (or model): Rounded off, the s-value s shows the number of heads in a row one would need to see when tossing a coin to get the same amount of information against the tosses being “fair” (independent with “heads” probability of 1/2) instead of being loaded for heads. For example, if p = 0.03, this represents $–log_2(0.03) = 5$ bits of information against the hypothesis (like getting 5 heads in a trial of “fairness” with 5 coin tosses); and if p = 0.25, this represents only $–log_2(0.25) = 2$ bits of information against the hypothesis (like getting 2 heads in a trial of “fairness” with only 2 coin tosses).
If we know the null hypothesis is not exactly true, yet the result is not statistically significant, then that is an issue of sample size, or statistical power. Statistical significance is not really a goal, it's a necessity that one achieves with appropriate statistical power. Given the same effect size, the results of two experiments can be statistically significant or not depending on the sample size. However, I trust the statistically significant effect size more than the other because it had a bigger sample size.
You can just state the result: "On average, Gurples were 10 cm taller than Cheebles (Difference in Height = 10 [5, 14]; mean, 95% CI, p=0.03)."
