Can I do a hypothesis test to see if two populations are different if there are known subpopulations within the data?

Question

I have a continuously variable output (diffusivities) that I have measured from two different populations (say "Case 1" and "Case 2"), and I am trying to see if the two populations are different. The trouble is, each population has two known subpopulations ("ventral" and "dorsal"), which we know have different diffusivities between them. So if I combine the data, Case 1 and Case 2 both have bimodal distributions. I would like to know if I can state that the differences behind the two Cases are driving differences in diffusivities as a whole.

If you would like, I can post more details about the actual problem, but the simplest analogy I could come up with was something like asking whether two sheep species have different weights. Both species are sexually dimorphic, so we know for sure that, on average, males weigh significantly more than females. I know we could test males and females separately, but I would like to see if species A weighs more than species B in some sort of "combined" sense. Maybe it's because I'd like to test whether genetic differences between the two species are the culprit, or maybe I need to increase the power of the test, but for whatever reason I would like to combine the two data sets and get a p-value for the combined population.

Would there be a simple (or even not-so-simple) way of doing that?

Thanks!

Answer 1

There are many help sites on the Internet for doing two-factor ANOVAs. I will leave that part to you. The rest of this is in case you find that residuals of your ANOVA are not normal (or differ widely in variance among the four treatment combinations).

Below is an example of two (simulated) bimodal samples, and a permutation tests to see if their means differ. The main idea is that we randomly reassign observations to groups 1 and 2, and find the difference d.prm in means for each. After 100,000 iterations we have a good approximation to the permutation distribution of differences. The P-value of the permutation test is the proportion of d.prms with absolute value exceeding the absolute value of d.obs, the difference in means for the observed data.

set.seed(2020)
x1 = c(rnorm(50, 104, 5), rnorm(50, 134, 5))
x2 = c(rnorm(50, 100, 5), rnorm(50, 130, 5))
a = min(x); b = max(x) # to put histograms on a common scale
par(mfrow=c(2,1))
 hist(x1, prob=T, br=15, xlim=c(a,b), col="skyblue2")
 hist(x2, prob=T, br=15, xlim=c(a,b), col="skyblue2")
par(mfrow=c(1,1))

The two samples suggest bimodal population distributions of about the same shape, but with different locations.

x = c(x1,x2);  g = rep(1:2, each=100)
d.obs = mean(x[g==1])-mean(x[g==2]);  d.obs
[1] 5.122707  # observed diff btw group means

Code for permutation test:

m = 10^5;  d.prm = numeric(m)
set.seed(728)
for(i in 1:m) {
 g.prm = sample(g)
 d.prm[i] = mean(x[g.prm==1])-mean(x[g.prm==2])
}
mean((abs(d.prm) >= abs(d.obs)))
[1] 0.02504    # P-value of permutation test

The P-value of this two-sided permutation test is the area in the tails of the histogram outside the vertical red lines.

hdr="Permutation Dist'n of Mean Differences"
hist(d.prm, prob=T, br=60, col="skyblue2", main=hdr)
 abline(v = d.obs, col="red", lwd=2)
 abline(v = -d.obs, col="red", lwd=2, lty="dotted")