0
$\begingroup$

Assume my hypothesis is: "Light supports the performance of students." The experimental setting consists of two room conditions that students are randomized to. The "light" group takes a test in a well lit room and the "dark" group has to take the same test in almost complete darkness (This sounds like a great idea for an experiment!). For each group, the test consists of 5 components: listening, speaking, presenting, translating and writing.

Surprisingly, the outcome is that the light experimental group outperformed the dark group in the total score. But it turns out that the results of the single test components differ to some extent; i.e. there is almost no difference as to listening, speaking, and presenting, but with regard to translating and writing there is some clear divergence. Below you find some numbers to help you imagine the situation.

example values

Assuming that the general probability value is p < .05 for this experiment, the p-values of the single components cannot be interpreted without an alpha correction due to multiple testing, as far as I know. Applying a Bonferroni correction creates a new p < .01, but then there is insufficient statistical evidence to reject the null hypothesis for the tasks of writing and translating.

So here is my idea: I would like to combine the two tasks to a cluster with the explanation that for both of them you need to be able to see what you are writing. In other words, I want to create a kind of a posteriori hypothesis: "Light supports the performance of students in written tasks."

In order to get statistical significance I would like to adjust the alpha value with the Bonferroni correction like this: alpha = .05/5*2

Does this makes sense and is it a sound procedure?

Many thanks for any advice in advance

$\endgroup$
2
$\begingroup$

You should be very wary of clustering and re-testing your comparisons based on the results of those comparisons. If you had observed that the speaking and listening tasks were uncorrected significant, and then lumped those two together to reduce your number of comparisons, you might get a significant result. You could do that with any pair of significant comparisons, but that doesn't make it appropriate. You're essentially selecting the significant results and clustering the others together, which is in effect pretending you didn't do the multiple different tests.

If you had come in with that hypothesis that light affects only written tasks a priori, it would be appropriate to test that even without multiple hypothesis correction. But you can't test your comparisons, find the marginally significant ones, and then change your hypothesis after the fact to better fit your data.

$\endgroup$
0
$\begingroup$

You don't want to want to get in to manipulating the analysis like this to make the results come out more to your liking. It is statistically wrong, and your audience will recognize it as shenanigans.

My advice in this case is to not use a p-value or alpha adjustment for multiple tests, and to accept that this means you may have an inflated probability of a type I error among the family of tests. You don't have a tremendous sample size, and this is a kind of preliminary study it seems, so you probably want to err on the side of not missing effects that may be there, even if you may have to accept some false positives as well. Deciding how to manage this kind of tradeoff is really the responsibility of the analyst.

I would also advise you to look at the Cohen's d. (I assume that's what in the d column). The effect size for Writing and Translating is rather large. Combined with the reasonably low p-values, this suggests that there is something worth noting in the results of these Tasks. Also consider the the absolute difference in means. For Writing it was 5 points, which seems considerable for these data.

The point is to let the data speak. There's no point in stifling it because someone told you are required to make a Bonferroni adjustment. You just must understand what the consequences are of not doing so: You risk an inflated type-I error rate. Sometimes that's okay, and sometimes that's something to be avoided.

$\endgroup$
  • $\begingroup$ Thanks everybody for all your help. I will follow your advise and focus more on descriptive statistics and the effect size. $\endgroup$ – pusete Jul 31 '18 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.