9
$\begingroup$

Some style guides instruct authors to report not only the results of a hypothesis test, but also the value of the test statistic from which it was calculated. For example, APA Style suggests t(DOF)=t statistic, p=p value for reporting the results of a t-test, as shown in the following examples, taken from here¹.

One sample: “Younger teens woke up earlier (M = 7:30, SD = .45) than teens in general, t(33) = 2.10, p = 0.31″

Dependent/Independent samples: “Younger teens indicated a significant preference for video games (M = 7.45, SD = 2.51) than books (M = 4.22, SD = 2.23), t(15) = 4.00, p < .001.”

The $p$-value is obviously useful², as it tells you if the results are unlikely under the null hypothesis. Descriptive statistics are important for understanding the characteristics of the data/subject pool, as well as the size of any purported effect.

The $t$-statistic (or similar for $\chi^2$ tests) seems more like an intermediate step, needed to get one from the other. Likewise the degree(s) of freedom are often closely related to the number of data points, but the actual value of $N$ seems easier to interpret.

How does including the test statistic help me interpret these results? Is it merely convention, an aid to meta-analysis, or can astute reader learn something from these numbers?

I'm particularly interested in the case of "simple" tests; I can imagine that $F(x,y)$ tells you something about the design of an ANOVA that might not be clear from a written description.


  1. These examples don't actually seem very good to me; the mean of the within-subject differences would be more informative, for example.

  2. In a NHST framework, of course. Assume we're happy working in it for the purposes of this question.

Improve this question
$\endgroup$
6
  • $\begingroup$ Controversial topic: Maybe an overdue attempt to squelch mindless use/abuse of P-values standing alone as 'evidence'--especially rampant in social sciences for some years. Recent position papers by ASA and others have pointed out inappropriate gaming of P-values leading to inappropriate, probably false or demonstrably irreproducible 'discoveries'. // One extreme example: You do complex study. Find several dozen P-values for various 'effects'. Report only the 5% of 'significant' ones--anticipated by chance alone. Don't mention that the rest of the study showed nothing. $\endgroup$
    – BruceET
    Commented Apr 21, 2020 at 19:06
  • $\begingroup$ I'm...not totally sure about that. I definitely agree those are issues but I'm not seeing how the t-statistic helps address them. I think this style also predates the those position papers (though concerns about NSHT have been around for a while). $\endgroup$
    – Matt Krause
    Commented Apr 22, 2020 at 16:01
  • 2
    $\begingroup$ In my experience one is happier following style manuals than trying to understand them. They are often arbitrary policies set without much scientific input or as a compromise after conflicting input. You can work through scientific societies to try for changes. // As to this particular issue, I see no problem in reporting sample sizes, t-values, and whether one or 2-sided test, in addition to P-value. I've heard few pubs have banned mention of P-values, which I think is going too far. (But if enough other info is revealed, reader can deduce P-values.) $\endgroup$
    – BruceET
    Commented Apr 22, 2020 at 17:21
  • 1
    $\begingroup$ @BruceET Sorry I may be hijacking the thread here. You say Report only the 5% of 'significant' ones--anticipated by chance alone. I understand why this is a problem. What I do not understand is why it is blamed on NHST, p-values or frequentist statistics in general. If you report only the extreme posterior distributions without mentioning all the others, wouldn't the problem of reproducibility be the same? $\endgroup$
    – dariober
    Commented Apr 23, 2020 at 14:31
  • $\begingroup$ If you test multiple hypotheses in the same data, you have to use a method of avoiding false discovery. If you are submitting a paper, follow the style manual. This site discourages chatting in comments. Done here. $\endgroup$
    – BruceET
    Commented Apr 23, 2020 at 15:59

1 Answer 1

Reset to default
2
$\begingroup$

I'm certainly not a mind reader as to why or how these recommendations came into being; however, I can at least speculate and share some personal experience in APA.

As you observed in your response to @BruceET, the APA guidelines for reporting the test statistic do predate the major position papers on the misuse of p-values. As such, the recommendations also predate the transition to assuming that effect sizes will be reported for every test. Prior to such a requirement, interested readers could at least compute some rough effect measures from the test statistics (e.g., d = t/sqrt(df)).

Ultimately, though, I think the short answer is as a matter of transparency. If someone reports their results as t(30) = 1.50, p < .001, d = 1.25, then readers (ideally this would be caught by the reviewers/editors) can look at those values and see clearly that this is incorrect. Similarly, if someone were to just report, "The means differed significantly, p < .001," then we may be missing some vital information. I think your question is fair in a world where research know exactly what they're doing and are making well-informed decisions; however, the reality is that statistical software packages can't always warn someone that they are requesting a non-sensical test. I think this also matters for questions of parametric versus non-parametric tests. Saying something like, "the variables were significantly correlated, p < .05" doesn't tell us the kind of correlation and thus might not help us understand the potential meaning of that result.

On the note of degrees of freedom, I do think part of that is related to ways of computing effect sizes, but I think it is also a transparency issue as well. Just because the total sample for a study may be large, once you factor in missing observations, the n for specific statistical tests and the N of the sample can be fairly different. Should an author find that their test was non-significant in the entire sample, they may be tempted to start looking at subgroups or portions of the sample, and reporting the degrees of freedom for each of those results helps prevent mis-interpretation.

In short, I think in a perfect world where everyone is very responsible in their research and everyone understand the statistics they are using, there wouldn't be any need for reporting the statistics themselves. I think, however, as a matter of pragmatism and good faith effort for transparency in research, reference styles like APA recommend such practices.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.