2
$\begingroup$

1) Should I be reporting the results of all the t-tests I ran, or can I just talk about the ones that were significant?

2) Must I report the t-statistic, df, effect size in all cases?

I am pressed for space, this is an extended abstract for a Computer Science HCI conference.

$\endgroup$
1
  • 1
    $\begingroup$ "Do I report them all" and "Do I report them all in an abstract" may well have different answers. $\endgroup$
    – Glen_b
    Commented Dec 25, 2014 at 15:03

2 Answers 2

5
$\begingroup$

As a minimum, you must report the number of tests you did. Only reporting the "significant" findings without mentioning the others is "P-hacking". It is dishonest and leads to misleading conclusions.

$\endgroup$
1
$\begingroup$

If you're just talking about what goes into your abstract, that's can be just the interpretation of your findings. You might not have room for any detailed statistics but do prioritize honesty of reporting of the methods and results. In doing so, the number of tests should have just as high a priority as what is significant and effect size.

When you report the full results at the conference you must report all of the tests you did. Feel free to highlight the ones that are important to your conclusions but note that the meaning of a test depends on other tests run in the same context and whether they're on the same data or not. It also depends on whether they're theoretically driven or not. Search for questions here and elsewhere on multiple testing or multiple comparisons so that you can interpret the tests correctly.

Also see this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.