2
$\begingroup$

I have done an analysis using SPSS. I would like to compare the situation before and after treatment in the same patient, so I used the Wilcoxon test (not normally distributed). Now I look at 3 different parameters in 3 regions of interest. I have a total of 9 p values as a result. Now I wonder if I should correct that p value for the number of outcomes, even in such a small and not normally distributed population? Then I would come to a cut-off value of 0.05 / 9 = 0.006 for a statistical significant result. Is this the right thing to do or is this not necessary in this case? Or can I just use a p-value of 0.05 as a cutoff point?

The p-value I get from it are: p = 0.003, p = 0.002, p = 0.008, p = 0.005, p = 0.002, p = 0.085, p = 0.001, p = 0.002, p = 0.044

Thank you if you know the answer, because it makes the interpretation of the results different.

$\endgroup$
1
  • $\begingroup$ It would be important to account for multiple tests in order to keep your false-positive rate at the level where you think it is ($\alpha$). However, Bonferroni is quite conservative (sucks away power to reject). Consider an alternative method, such as Bonferroni-Holm. $\endgroup$
    – Dave
    Commented Feb 19, 2020 at 14:17

1 Answer 1

1
$\begingroup$

Bonferroni correction is not about sample size, but about the number of tests. Look at the rationale for applying the Bonferroni correction: If I have a single test, then the chance of having a false positive is $\alpha$. In other words, even if the null hypothesis $H_0$ is true, you will reject it in a ratio of $\alpha$ cases. If you now use two tests, e.g. with different parameters, then you have two null hypotheses (let's call them $H_0^1$ and $H_0^2$), each of which is rejected falsely with chance $\alpha$. So the probability of rejecting any of these two is $1-(1-\alpha)^2$. So you have a higher chance of a false positive. Normality or sample size does not factor into this consideration.

There is an important point, considering the actual $H_0$ in question: When would you reject this overall $H_0$? Do you reject it when ANY of the actual tests are significant (use Bonferroni), or do you reject it when ALL are significant (do not use Bonferroni)? With three parameters and three ranges, there can even be combinations, such as reject the $H_0$ if ANY of the ranges for ALL of the parameters are significant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.