# Detect if there is actually two populations in a sample

I have been counting stomata on fossil leaf material to apply a known relationship between stomatal index and CO2. I thought that the material was all from one population (one species at a given site). However, exploration of the data suggests there may be two populations. I interpret these to be the species I was targeting and a hybrid, which are difficult to distinguish by leaf morphology (For reasons of stratigraphy we can rule out that these were actually two different times and thus different 'real' CO2 values).

I have been able to find information on how to determine if two samples are from different populations, but not if you took one sample and seem to have two different populations. Would it be acceptable to divide the distribution (say split it at 6.5) and use a Wilcoxon-Mann-Whitney test to determine if two samples are significantly different?

What is an unbias way to determine if these really are two populations?

These are the stomatal index results for the 41 leaves.

[1] 5.172414 5.246914 5.276382 5.278592 5.288462 5.306122 5.323194 5.325444 5.357143 5.366726 [11] 5.367232 5.376344 5.384615 5.504587 6.053269 6.854839 6.910569 7.006369 7.036247 7.112069 [21] 7.156673 7.231920 7.311828 7.416268 7.440476 7.448494 7.491857 7.526882 7.526882 7.534247 [31] 7.547170 7.559395 7.605634 7.671233 7.749077 7.925408 7.964602 8.064520 8.247423 8.252427 [41] 8.436214

Let's start with terminology. Population in statistics is the "set of entities under study". When designing the study, we define the population of interest and then draw samples from this population. So sample cannot "consist" of multiple populations. More appropriate wording would be to talk about "groups", "clusters", or "subpopulations".

To find clusters in your data, you could use algorithms, that will try to split your data into a predefined number of groups, given such criteria. Usually we are aiming at the samples within each cluster being most similar to each other, while the clusters most dissimilar. Notice the logical problem in here: if you would first group stuff in such way that the groups are dissimilar from each other and then test if they differ, then this gets circular. If your test fails, maybe the clustering algorithm was not good enough, or test not sensitive enough? It opens many ways to "torturing the data until it confesses" and generally is a bad idea.

One approach that can be justified, is to use model-based clustering (i.e. model, as mentioned in the other answer by Stephan Kolassa) with one, or two clusters and then conduct a likelihood-ratio test to compare two models. If the data is more "likely" given the two-cluster model, then you can say that two-cluster solution "fits better" the data, though it doesn't prove that there were actual subpopulations. This approach would need you to be able to define a statistical model that describes the data, so it is more complicated then using "black box" clustering algorithm.

• I guess I was mentally caught up in the population I meant to study vs the population I have studied. Thank you for the wording suggestions. It will take me a while to read through all this, but I am onto it - too newb to upvote! Jun 9, 2020 at 10:20
• I don't quite see why this was downvoted. +1 from me. Jun 9, 2020 at 11:28
• I chose this over @StephanKolassa because of the extra detail, but thank you to you both. Because a normal distribution would be expected in this system I used a gaussian-mixture model approach and compared by the likelihood ratio test. For all the reasons laid out, this showed the 2 cluster model fitted better. I then assumed those were two seperate clusters, and tested if other characteristics of the leaves were also significantly different between the clusters using t-tests (as the data matched the assumptions). There was also a significant difference in leaf size so an interesting result! Jun 15, 2020 at 1:36

There is no way to do this by non-parametric paradigm, just think of it: the sampled distribution is a completely legit one, there is nothing preventing a single-population distribution from having two separate high density zones.

But if you turn to parametric models, you may assume that your sub-populations are gaussian, and gaussian distribution has only one bell-shaped high density region. If you do so, you can run EM clustering to estimate the likelihood of a mixture model of two gaussian clusters, and compare it to the one-population scenario with a likelihood ratio test.

Looking to your data, this test will certainly show high significance. But there are problems:

• EM clustering tends to inflate likelihood of multiple sub-populations hypothesis when the real distributions are not quite gaussian
• even more importantly, performing a test on a hypothesis formulated after looking at the data gives auto-confirmation bias.

In brief, I recommend you to let it go, and just comment the observed distribution as "likely coming from distinct sub-populations", or something around this line. Any test about it would be biased and unreliable.

• Thank you I have two methods for determining CO2. One, over the whole data, gives values 380–500, the other, 260–420. The existence of two different equations for the same species, which both seem to work, is an outstanding problem in my field. If I do have two seperate clusters, say one is a hybrid and the other the target species, and I apply the two methods, one cluster using method A suggests a range of 390-440, and the other cluster using method B a range of 395-421. My sample is too small, of course, but it could be an important observation to guide future work. Jun 9, 2020 at 10:16
• I am not sure I understand what you are asking in this comment. If it is a different problem than the one your original question is about, maybe post a new one providing some more detail. Jun 9, 2020 at 10:39
• Sorry Carlo. I think I overlooked/didn't absorb your second last sentence properly. I was explaining why it is important/necessary to point out the likely separateness of the two clusters, but that is unrelated to the point you are making about auto-confirmation bias. I think I understand, rereading you comment today. Thank you. Jun 10, 2020 at 1:23
• I ended up using a clustering and test approach but then using the model selected clusters to test if other characteristics of the leaves were also different between the two groups and did find that the shape/size of the leaves differs significantly in a way that was not obvious to look at and may be useful in determining what is going on. My write up will still address the issue of confirmation bias though and it did put the 'middle' value in a cluster I wouldn't have by eye. Jun 15, 2020 at 1:47
• that's another problem: if you take two sub-samples created by clustering, there is no point in testing them for mean difference, the result will always be significant if you have enough data, because clustering doesn't allow them to be mixed. Jun 15, 2020 at 7:24

In statistical terms, you are wondering whether your data comes from a mixture of two (or more) populations, as against coming from a single population. Looking at the or more specifically the tags will be helpful. Number of components for Gaussian mixture model? includes a very good approach for deciding between one or two components based on comparing likelihoods.

• Thanks @Stephan Kolassa! It will take me a while to get through reading up on it, but you have scheduled my tomorrow for me. I will come back to confirm when I get it working. Jun 9, 2020 at 9:52

Other answers have discussed clustering, which is appropriate here. Let me briefly discuss the Wilcoxon-Mann-Whitney test. Basically, the MW test assesses if values in one group tend to be higher than the other (see my answers here or here). That is, if you picked a number from one group and a number from the other group, would the first typically be larger? If you split your data into higher than some cutpoint and lower than the cutpoint, the answer will always be yes by design. The question of whether the MW will be significant is a question of power. If you have at least 4 data in each group, then a MW run over the data will always be significant. In short, the procedure you have in mind will 'work' in the sense of giving you a significant result, but it won't be telling you what you want to know. For an example of Gaussian mixture modeling, tested with the parametric bootstrap cross-fitting method, see my answer here: How to test if my distribution is multimodal?

• Thank you for explaining this. It was my gut feeling that this would be a kind of self fulfilling prophecy, if you will, but I appreciate you taking the time to explain why. Jun 15, 2020 at 1:39
• You're welcome, @The_Tams. Good luck with your project. Jun 15, 2020 at 4:32