0
$\begingroup$

I have 4096 variables from two groups. I want to know which of them have means that are significantly different between the two groups. So I generated the following numbers for each of them:

  1. p-value of Shapiro Test for Group 1 for normality
  2. p-value of Shapiro Test for Group 2 for normality
  3. p-value of F-test for similar variance
  4. p-value of two-sample t-test for the case when both are normal and similar variance
  5. p-value of two sample Welch test for the case when both are normal and different variance
  6. p-value of Wilcoxon Rank test for the case when one of them is not normal

So I have 4),5),6) for a total of 12,288 p-values to compare. Some of the data are not normal and/or different variance. Can I adjust all the 12,288 p-values together using the Benjamini-Hochberg procedure?

Or should I only consider Wilcoxon Rank Test p-values because not all of my values are normal and only apply BH procedure for them?

$\endgroup$
1
  • $\begingroup$ Whoever used Wilcox as a shorter name in R to save two characters did the community a disservice. The test in question was first thought up by Frank Wilcoxon. (Bizarrely later he did have a co-author Wilcox, but Wilcoxon is the name you need.) $\endgroup$
    – Nick Cox
    Commented Apr 14, 2021 at 8:39

1 Answer 1

1
$\begingroup$

To start off with, I will assume this is a randomized experiment (or some other setting where directly comparing the two groups makes sense). For example, what you should not be combining is a question of "Does this blood pressure drug work?" with an analysis that compares people who happen to be prescribed blood pressure drugs with those who are not prescribed such drugs. Simply doing a t-test in that setting mixes the questions of what the drugs do with the question of whether the two populations of people differ.

First, that sounds like a huge number of analyses and rather like an extremely exploratory setting. Whether null hypothesis testing really makes sense in this setting seems debatable to me. As always, it's important to keep in mind that what is flagged as "statistically signficant" due to the random variable that is the p-value being below some threshold does not equate to either a) the rejected null hypothesis definitely being untrue or b) the difference between the groups being of practical significance (in the traditional sense of the word). And the other way around, a p-value above some threshold does not mean that there is no difference between the groups (or that any "non-detected" difference between the groups is not of practical significance).

Second, a procedure that consists of

  1. testing assumptions of such as normality of residuals (I'm not sure why you would test the normality of the distribution in each group - looking at regression residuals seems more standard and generalizes to more situations) or homogeneity of variances for a regression model (or t-test), and
  2. switching between different tests based on whether there's a "significant" deviation from your assumption

invalidates the p-value you get from the test you pick in step 2. Thus, that tends to be a problematic approach. The best situation is if you can choose your test based on subject matter knowledge (e.g. if you were to analyze blood pressure differences between two drugs, then we just know that assuming approximately normally distributed residuals is okay).

Third, one way to avoid the problem above is to indeed just always do a rank-test. Theoretically, they are pretty okay in terms of efficiency for decent sample sizes. However, in practice we often have additional information (e.g. in a randomized experiment measurements before applying the randomized intervention), which we want to put as a covariate into a regression model (that at least takes you away from the simple Wilcoxon-rank-sum-test).

Once you have meaningful p-values for each analysis, you can then perform some multiplicity adjustment, if that really makes sense.

For alternative approaches, it's worthwhile to look at the literature for other areas where mass screening of data are done (e.g. genome wide association studies, mass screening candidate drug substances with in vitro assays), to think about exploiting correlations in the data (e.g. if some measurements are from the same experimental units and others not, perhaps some variables are more related to each other than others?), hierarchical Bayesian models (e.g. using the regularized horseshoe prior) and there's probably a bunch of other things that might possibly be relevant.

$\endgroup$
1
  • $\begingroup$ Excellent advice, and it's not contradicting it really to suggest getting bootstrapped confidence intervals for the differences between the means and avoiding this entire fandangle. $\endgroup$
    – Nick Cox
    Commented Apr 14, 2021 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.