I have a continuously variable output (diffusivities) that I have measured from two different populations (say "Case 1" and "Case 2"), and I am trying to see if the two populations are different. The trouble is, each population has two known subpopulations ("ventral" and "dorsal"), which we know have different diffusivities between them. So if I combine the data, Case 1 and Case 2 both have bimodal distributions. I would like to know if I can state that the differences behind the two Cases are driving differences in diffusivities as a whole.

If you would like, I can post more details about the actual problem, but the simplest analogy I could come up with was something like asking whether two sheep species have different weights. Both species are sexually dimorphic, so we know for sure that, on average, males weigh significantly more than females. I know we could test males and females separately, but I would like to see if species A weighs more than species B in some sort of "combined" sense. Maybe it's because I'd like to test whether genetic differences between the two species are the culprit, or maybe I need to increase the power of the test, but for whatever reason I would like to combine the two data sets and get a p-value for the combined population.

Would there be a simple (or even not-so-simple) way of doing that?


  • 1
    $\begingroup$ Starting with what may be the simplest approaches: (1) You could try a two factor ANOVA with factors Case (levels 1 and 2) and Location (levels V and D). You would be mainly interested whether Cases 1 & 2 differ, but Location and Interaction effects might provide additional useful insights. You would want to check whether residuals from the model are normally distributed. (2) If not normal, but bimodal distributions for Cases 1 and 2 are roughly of the same shape, you might try a permutation test of Case 1 vs. Case 2. $\endgroup$
    – BruceET
    Jul 28, 2020 at 21:16
  • $\begingroup$ Thanks @BruceET! I thought about two-way ANOVA, but I didn't really understand that it could be what I was looking for until you pointed it out. I appreciate the help for such a naive question! $\endgroup$
    – thedu
    Jul 28, 2020 at 22:04

1 Answer 1


There are many help sites on the Internet for doing two-factor ANOVAs. I will leave that part to you. The rest of this is in case you find that residuals of your ANOVA are not normal (or differ widely in variance among the four treatment combinations).

Below is an example of two (simulated) bimodal samples, and a permutation tests to see if their means differ. The main idea is that we randomly reassign observations to groups 1 and 2, and find the difference d.prm in means for each. After 100,000 iterations we have a good approximation to the permutation distribution of differences. The P-value of the permutation test is the proportion of d.prms with absolute value exceeding the absolute value of d.obs, the difference in means for the observed data.

x1 = c(rnorm(50, 104, 5), rnorm(50, 134, 5))
x2 = c(rnorm(50, 100, 5), rnorm(50, 130, 5))
a = min(x); b = max(x) # to put histograms on a common scale
 hist(x1, prob=T, br=15, xlim=c(a,b), col="skyblue2")
 hist(x2, prob=T, br=15, xlim=c(a,b), col="skyblue2")

The two samples suggest bimodal population distributions of about the same shape, but with different locations.

enter image description here

x = c(x1,x2);  g = rep(1:2, each=100)
d.obs = mean(x[g==1])-mean(x[g==2]);  d.obs
[1] 5.122707  # observed diff btw group means

Code for permutation test:

m = 10^5;  d.prm = numeric(m)
for(i in 1:m) {
 g.prm = sample(g)
 d.prm[i] = mean(x[g.prm==1])-mean(x[g.prm==2])
mean((abs(d.prm) >= abs(d.obs)))
[1] 0.02504    # P-value of permutation test

The P-value of this two-sided permutation test is the area in the tails of the histogram outside the vertical red lines.

hdr="Permutation Dist'n of Mean Differences"
hist(d.prm, prob=T, br=60, col="skyblue2", main=hdr)
 abline(v = d.obs, col="red", lwd=2)
 abline(v = -d.obs, col="red", lwd=2, lty="dotted")

enter image description here

  • $\begingroup$ Thanks again, this is great! $\endgroup$
    – thedu
    Aug 1, 2020 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.