2
$\begingroup$

I have a set of 56 variables, which are functional connectivity values, for two groups (control_group vs. patient_group) between two phases (pre-phase vs. post-phase). I have a small sample size (9 vs. 9). The variables are correlated between each other, because they are extracted from areas of the brain close to each other. I performed Mann-Whitney tests to see if there are any differences in the values of each variable, between the two groups, for each phase of the experiment. Again, I performed a Wilcoxon test to see if there are differences within the same group between two different phases for each variable.

Is it possible to perform a p-value correction for multiple comparison, and most importantly is this a situation that requires such a correction?

EDIT: Here's my implementation of the p-value correction.

test_results <- data.frame(Variable = character(0), W_statistic = numeric(0), p_value = numeric(0))
for (col in colnames(pre_data)[-c(1, 2)]) {
  pre <- wilcox.test(pre_data[[col]] ~ group, data = pre_data, paired = FALSE)
  result <- data.frame(Variable = col, W_statistic = pre$statistic, p_value = pre$p.value)
  test_results <- rbind(test_results, result)
}
test_results <- data.frame(Variable = character(0), W_statistic = numeric(0), p_value = numeric(0))
for (col in colnames(pre_data)[-c(1, 2)]) {
  pre <- wilcox.test(pre_data[[col]] ~ group, data = pre_data, paired = FALSE)
  result <- data.frame(Variable = col, W_statistic = pre$statistic, p_value = pre$p.value)
  test_results <- rbind(test_results, result)
}
p<-test_results$p_value
pval <- p.adjust(p, method = "HR", n=length(p))

However all the p-values went up terribly

test_results$p_value
 [1] 0.64395100 0.52510244 0.22504302 0.32603276 0.72886457 0.38616628 0.72886457 0.81724421 0.29838222 0.86240136 1.00000000 0.95392967 0.86240136 0.81724421
[15] 0.64395100 0.04949851 0.52510244 0.90801281 0.81724421 0.45262538 0.90801281 0.25961381 0.60309532 0.72886457 0.38616628 0.86240136 0.86240136 0.72886457
[29] 0.68591197 0.24790442 0.20372873 0.13307370 0.60309532 0.90801281 0.41861982 0.27234354 0.32603276 0.29838222 0.60309532 0.68591197 0.68591197 0.56344833
[43] 0.90801281 0.86240136 0.77268577 0.56344833 0.60309532 0.81724421 0.77268577 0.60309532 0.86240136 0.68591197 1.00000000 0.45262538 0.95392967 0.45262538
> pval
 [1] 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 1.0000000 0.9892604 0.9778599
[14] 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599
[27] 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599
[40] 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599 0.9778599
[53] 1.0000000 0.9778599 0.9892604 0.9778599
$\endgroup$

2 Answers 2

1
$\begingroup$

Yes, a scenario where you are performing 56 different comparisons (or twice that) is one that would usually require a multiplicity correction. With a per-test type I error rate of 5% you will now have a ~94% chance to make a type I error instead. You are in fact controlling the false discovery rate (intuitively, accepting up to $\alpha$ of your tests to be false positives) and not the family-wise error rate (ensuring at most one false positive with probability $\alpha$), which would be an even stricter procedure. With $n=18$ and 56 features you'd need some pretty big effect sizes to achieve any statistical power here.

I think a bigger problem is that you don't seem to be testing the relevant effect though: a well-designed experiment should have the two groups be as exchangeable as possible in the 'pre' (intervention?) phase. Furthermore, the difference between pre and post within each group is not what is relevant to your intervention, it is rather the difference across groups in change from pre to post. I would subtract (if that makes sense) each subject's post from pre and compare these changes across the two groups.

Edit: to expand on why the P-values change when you put them through p.adjust, recall what the Benjamini-Hochberg method does to your $m$ ordered P-values:

  1. For your given false discovery rate $\alpha$, find the largest $k$ so that $P_k<\frac{k}{m}\alpha$.
  2. Reject $H_0$ for all tests up to $k$ inclusive.

You have $m=56$ and the lowest P-value is about $0.0495$, but per above it should have been below $\alpha / 56\approx0.0009$ to be rejected at $\alpha=0.05$. Proceeding upwards like that you'll see that none of the P-values meet this threshold.

What p.adjust returns is sometimes called a q-value, which is essentially the $\alpha$ you would have to accept to reject that particular hypothesis. Going through possible values for $\alpha$ you'll see that the very first rejection of any hypothesis in your family happens at $\alpha\approx 0.9779$, which manages to reject the 52nd smallest P-value of about $0.908$ as this is ever so slightly less than $\frac{52}{56}*0.9779$ (not necessarily in the numerical precision I'm showing here). This means that all tests up to the 52nd are in one blow rejected... at $\alpha=0.9779$. The last few tests follow the same principle but would require even higher FDR to reject, culminating at 1.

$\endgroup$
2
  • $\begingroup$ Thanks, @PBulls. I'm having trouble understanding why the adjusted p-values are mostly similar and why the increase in the p-value does not seem consistent with the uncorrected p-value. $\endgroup$
    – Ed9012
    Commented Nov 15, 2023 at 11:22
  • $\begingroup$ I've edited some explanation on that into my answer. $\endgroup$
    – PBulls
    Commented Nov 15, 2023 at 14:04
0
$\begingroup$

The 'corrections' for multiplicity always entail a loss of power to detect interesting effects and that's why your post-'correction' p-values are large. The mindset that adjustments for multiplicity are always appropriate or necessary is wrong and damaging. They are only sometimes helpful and they are particularly harmful in preliminary research and hypothesis generating studies.

The desirability of multiplicity adjustments is very context dependent and so it cannot be decided only on the basis of the data and analysis arrangements. You do not say what type of study you are doing, what types of inference you might make, and what you might do next. In that circumstance it is not possible to directly answer your question of whether the situation requires a 'correction'.

There are many, many questions and answers already on this site that will help you understand the issues. Here are a few: How many p-value observations do you think are required before doing FDR correction

p value correction in multiple outcomes study

Correction for multiple comparisons

Do I need a Bonferroni correction?

You may also want to form a better understanding of p-values, evidence and error rates. For that I recommend A reckless Guide to P-values: local evidence and global errors

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.