Multiple paired t-tests. Bonferroni number of comparisons?

Question

I have done 9 paired t-tests, to see if there are significant changes in pre and post measurements of three variables (FFMQ-SF, GSE, WEMWBS), across three groups (Engagement level: minimum, medium, maximum). See the attached table.

I believe that to protect from Type 1 errors, I must conduct a bonferroni correction in which the critical significance level of .05 is adjusted. Question 1: Do I need to do this?

To perform a Bonferroni correction, one should divide the critical p-value by the number of comparisons being made. But I don't know whether I should divide 0.05 by 3 or by 9. I conducted 9 t-tests, but 3 for each scale/group. Question 2: what is the correct option?

Question 3: Finally, am I correct in thinking that I should convert my p-values into adjusted p-values by multiplying by either 3 or 9?

Answer 1

Question 1: Yes and no. Yes, you should account for multiple testing but you do not have to use the Bonferroni correction. The Bonferroni correction is easy to apply but known to be very conservative - meaning that it often will not detect non-zero effects even though there are non-zero effects.

The Holm procedure is slightly more powerful than the Bonferroni correction and works under the same set of assumptions.

In addition, there are a number of Bootstrap based "step-down methods" that can be even more powerful. These methods use simulation techniques to find out about the correlation of the tests.

In your case, you may have some prior information about the correlation of the tests. For example, if groups are independent then applying a test to three different groups gives three independent tests. You can use that.

Question 2: 9 for the Bonferroni correction that you are suggesting.

Question 3: I find it best to report the unadjusted p values and only change the significance indicator (the "stars"). This way the reader has all the information to apply, for example, Holm instead of Bonferroni.