# Testing if two distributions have the same mean by using a sample distribution

ORIGINAL Phrasing: Let's say I have a population distribution, A. For example, this could be the distribution of the maximum height (not the current height) that someone will reach during their life. Let's say A only contains data for people born in 1900 and 1901.

Now imagine a subpopulation of this distribution, B. This distribution may or may not have the same statistics as A - I don't know. For example, B might be all of the Males in A, which might have a higher mean. Or it might be all of the people born in 1900, which I would not expect to be very different from A in terms of statistics.

Now let's say I draw a random sample from A such that it contains only members of B. So for example either all males, or all people born in 1900. It's assumed that I don't know everyone's gender or birth year in A, so I can't actually construct B and look at it.

How can I use my sample to determine if B's statistics (e.g. mean) differ from A? It seems like standard C.I. / p-value approaches can work. But at the same time I'm not sure, because the random sample of B may by chance not be representative of B, especially if the sample is small. In other words, B itself may have its own (different) variance and mean than the sample. So for example, if I did have B, I could compute C.I. with its statistics to determine if its mean is the same as A, but the C.I. computed from the sample will be different than those computed from B itself. And then there's the fact that B itself is also a subset of A.

Am I overthinking things? Are standard C.I. / p-value approaches sufficient?

EDITED Phrasing: Let's say I have a population distribution, A with size NA = 1000000. For example, this could be the distribution of the maximum height (not the current height) that someone will reach during their life. Let's say A only contains data for people born in 1890 through 1910. We don't actually have the birth year for any person, but we know it is in that range.

In theory, there is therefore a subpopulation, B, with size NB, of all people in A born in 1890, but this population cannot be observed, and NB is not known.

Now someone comes along and labels NC = 1000 people as being born in 1890. We'll consider these people sample C. He says that they were randomly selected from the NB people in B to be labeled (somehow, this person knew everyone who belonged in B, even though I don't). So far these are the ONLY people whose birth years I know.

Can we determine if B's heights are different from the total population A using the sample C, and put confidence on that? As a secondary followup, what if I can't assume that C is a random sample from B? What if someone just gives me a bunch of 1890 labels but I'm not sure whether that was randomly done or not?

My concerns are that there's two types of error going on. The error in B itself when comparing its mean to that of A, which I can't really observe, and the error in the fact that C, even if randomly sampled, can be a bad sample of B.

The other issue is that C (and B also) is part of A, so in comparing C to A, there's duplicate observations that belong to both distributions.

Am I overthinking things? Are standard C.I. / p-value approaches sufficient?

Problem: Consider a population of $$N$$ people, with each person $$i$$ having a characteristic vector $$(X_i, S_i)$$ which consists of a characteristic of interest $$X_i$$ and a binary covariate $$S_i \in \{ 0,1 \}$$. Let $$N_* \equiv \sum_{i=1}^N S_i$$ be the size of the subpopulation of people with $$S_i=1$$, and suppose that we are interested in comparing the population to the subpopulation. Without loss of generality, we can order the indices so that the first $$N_*$$ people are in the subpopulation, and the remaining $$N-N_*$$ people are not in the subpopulation.$$^\dagger$$

The goal of the analysis is to compare the characteristics of the population and the subpopulation. For definiteness, we will assume that we wish to compare the means of these groups. The means of the population, subpopulation, and non-subpopulation, are denoted respectively by:

$$\bar{X}_N = \frac{1}{N} \sum_{i=1}^N X_i \quad \quad \quad \bar{X}_{*} =\frac{1}{N_*} \sum_{i=1}^{N_*} X_i \quad \quad \quad \bar{X}_{**} =\frac{1}{N-N_*} \sum_{i=N_*+1}^{N} X_i.$$

Now, if I understand your description of your sampling correctly, you are sampling from the whole population via simple-random-sampling without replacement (SRSWOR), so you will get people from the subpopulation and the non-subpopulation at random. (This is a little ambiguous - at one point you say you sample from the population so that you only get values from the subpopulation, but at another point you say you can't identify the covariate until after sampling, which would mean that it is impossible to sample only from the subpopulation.) You want to use this data to test whether the mean of the population differs from the mean of the subpopulation. That is, you want to form a confidence interval or hypothesis test on the (unknown) value $$\bar{X}_N - \bar{X}_{*}$$.

Solution: If that is a correct interpretation of your sampling scheme then this problem can be handled using ordinary methods for comparison of two groups (the subpopulation and the non-subpopulation). To see this, it we note that with a bit of algebra it have be shown that:

$$\bar{X}_N - \bar{X}_{*} = \frac{N-N_*}{N} \cdot (\bar{X}_{**} - \bar{X}_{*}).$$

Thus, comparing the mean of the population and subpopulation is equivalent to comparing of the mean of the subpopulation with the mean of the non-subpopulation. (For a test of equality these are exactly the same, and for a confidence interval for the difference, you apply the scalar multiplier in the formula shown.) Comparison of these two groups is the famous Behrens-Fisher problem which is a well-known problem in statistics. The comparison is usually undertaken using a confidence interval or hypothesis test for the mean difference via Welch's approximation.

UPDATE: From the updated description of your sampling mechanism, it now appears that you are saying that you only have sample data from the subpopulation of interest, but you do not have any sample data from the remaining part of the population. If that is the case then you can estimate the mean of the subpopulation, but there is no basis to estimate the difference in means between the subpopulation and the rest of the population. If you would like to accomplish the latter then you will need some data (or some known summary statistics) from the remaining part of the population.

$$^\dagger$$ That is, we take $$S_1 = ... = S_{N_*} = 1$$ and $$S_{N_*+1} = ... = S_{N} = 0$$.

• As an example of the sampling: Imagine one main population of all people. There's a theoretically well-defined subpopulation of all males that can't be empirically observed. I know the gender of only a small fraction of all males, the all known males sample, and not of any of the other males or females. – CHP Oct 16 '18 at 22:01
• So then the question is, in determining if all males mean is different than all people, should my confidence interval capture the fact that there's a chance that all known males could be a bad sample of all males, even if randomly sampled from all males? What's the best I can do to determine if all males is different than all people, given the sample of all known males? My confusion is basically that there's two levels of sampling going on: All to All Males (biased sample), and All Males to All Known Males (random sample). And I'm not sure how to combine the error. – CHP Oct 16 '18 at 22:02
• Sampling error under SRSWOR is already taken into account by standard confidence intervals for this problem, so the possibility of a bad sample is already built in. Since you are sampling at random without knowing sex in advance, you should obtain a random sample of males and a random sample of females. You can compare these using standard methods for comparison of means in two populations. – Ben - Reinstate Monica Oct 16 '18 at 22:26
• In this scenario, I don't know who is a female. The only group I know is the comparison group. E.g. the all known males group is made up of only males. I don't know the group of any of the rest of the population. It's more of a One vs. All comparison. all males vs. all people comparison, using all known males as a proxy, to compare the mean of all known males to all people, in an attempt to conclude something about all males. – CHP Oct 16 '18 at 22:30
• Okay, I'm really confused now. I'm afraid I really can't follow your description of your sampling method; there are several things you are saying that seem contradictory to me. Perhaps you could give an update on your question with a detailed account of exactly how you are getting your sample. – Ben - Reinstate Monica Oct 16 '18 at 22:35

Well, here is one way. Males are approximately 4 inches taller than females. Male heights and female heights tend to be normally distributed. For your "A" population, fit the data with a mixture model consisting of two normal distributions. That will yield (1) a fraction of individuals with normally distributed shorter heights, which are likely to be female, and, (2) an another normal distribution with taller heights. Now the taller peoples' normal distribution with have a mean and standard deviation, that mean and standard deviation can then, for sufficiently large populations, be tested by Fisher's z test, against your B population of males. A finding of no significant difference should be associated with a confidence interval for the mean difference, which will give you some idea of how much uncertainty (e.g., how powerful) that negative result is.

BTW, to maintain independence, the "A" population should not have any sex related information per se.

For the second problem, 1900-1901 versus 1900 one needs the two data lists or it is intractable. If yes, we can eliminate the 1900 data from the 1900-1901 list to make two independent lists; one for 1900 and the other for 1901. Once the correction of the data is completed, if there is no subject overlap, the analysis itself could be done with either a CDF empirical distribution comparison or an explicit double mixture distribution independent male/female double comparison. However, the interpretation of the results depends on what the data is. When were the heights measured? At birth, now, at age 60?

Response to edit: One would use a find distribution routine for mixture models (similar to hidden Markov model) using a criterion like BIC for model selection. That will identify the number of distinct populations. The parameters of the selected model are then refined using a find distribution parameters routine. It does not matter how many "genders" one imagines to be present, the modelling will only return contributory results, which then have to be identified as particular sub-populations by other means. Mixture models will identify how many individuals are in each sub-population. It will not identify which data entries belong to which population in the region where the populations overlap, but it will yield how many there are in each population, and what the distribution parameters are for each sub-population.

Depending on what you are modelling, a normalization step, such as data ranking may be needed prior to other analyses. Also, two-step cluster analysis with k-means may be the starting point applicable depending on the exact circumstances of your modelling, and that may identify which data entries belong to which cluster.

• The problem in the question is a toy problem that mirrors a more complicated real life problem, so the specifics (e.g. height measurement dates) don't come into play. This solution is an interesting approach for the simplified version of the problem in the question, but in practice there may be like 20 groups or some other unknown number. At any given point I want to do a 'One vs. All' type of comparison. E.g. is Group 17 > All groups? Then later, is Group 3948 > All groups? – CHP Oct 16 '18 at 22:14
• @CHP It is not clear what you are asking. I can only answer what you do ask or the answer is "see all statistical methods," which would not be helpful. If you would add an edit to your question and ask a more precise question, including data types, distributions expected if known, and so forth, that would be more likely to elicit an answer that is more relevant to your needs. Not for naught the old expression is, "garbage in---garbage out." – Carl Oct 16 '18 at 22:30
• I'll try to re-word the question. But to give a closer example to the actual problem, I have a population of P = 100000 people. Someone says "These K = 1000 people are all males, and there's M total males in the population of P, where M is >> K, but is unknown. Tell me if the mean height of all of the M is > than the mean height of the population of P people as a whole, using the K people you know are males. You don't know the gender of the P - K group, and there could be an unknown number of genders, not just two." – CHP Oct 16 '18 at 22:39
• @CHP That does not change the form of my answer. One would use a find distribution routine for mixture models (similar to hidden Markov model) using a criterion like BIC for model selection. That will identify the number of distinct populations. It does not matter how many "genders" one imagines to be present, the modelling will only return contributory results, which then have to be identified as particular sub-populations by other means. Mixture models will identify how many individuals are in each sub-population. – Carl Oct 16 '18 at 22:48
• @CHP con't. It will not identify which data entries belong to which population in the region where the populations overlap, but it will yield how many there are in each population, and what the distribution parameters are for each sub-population. – Carl Oct 16 '18 at 22:51

I think the way you're thinking about this is a bit confused, let's analyse a few of the things you say:

Now let's say I draw a random sample from A such that it contains only members of B

If you draw a truly random sample from A, then given you know how A is distributed, you know the true distribution of the mean of the random sample. If you draw a sample from A such that it contains only members of B, and B turns out to be a biased sample of A, then it's usually pretty unlikely you were genuinely sampling at random from A.

This is a classic problem in, for example, election polling. Pollsters try to sample from the voting population (voting distribution A), but reach a biased subset (e.g. digitally and politically engaged urban voters, or B). They make inferences about the general population based on their sample assuming random error, but their sampling method means they were not randomly sampling in the first place so they get their inference wrong.

I think what you might be trying to ask, or at least a (albeit perhaps not the only) sensible question to ask, might be something like the following:

I draw a sample from A, and I want to know whether my sampling method is truly random, or whether I've obtained a biased subset.

(note that if B is sampled from A, but you find that "B's mean differs from A", this is equivalent to saying that the sample was not truly random in the first place).

One way of doing this is by calculating the p-value of obtaining the sample mean you obtained. You do need to know the details of the population distribution A you were sampling from, it isn't clear to me either from your question whether this is known or not.

Likewise, the Bayesian method would be to assume an underlying distribution for B. If you know that A is normally distributed with known mean and variance, you could assume that B is also normally distributed with the same variance, assume a prior which is centred around the mean of A, and do an inference on the mean of B using Bayes Theorem. Then you could calculate the confidence you have that the mean of B is within $$\pm x$$ of the mean of A. If your confidence is high that the mean of B within $$\pm$$ of the mean of A, where x is pretty small, you can conclude that your sampling method is likely to be "not very biased". If your sample sizes are small, then this method becomes very sensitive to your prior, something some people find dis-satisfactory about Bayesian methods.

• That's close. What I'm trying to ask is, is B truly different than A. I KNOW the sampling method is biased relative to A, but we can assume it's random relative to B. So the members I sample from B are randomly sampled from B, but B itself may or may not turn out to be like A. In my example, what I would be trying to determine is: "Are the population of M males actually taller than the N person population as a whole, assuming I ONLY know that these K people are males, where K << M, and I don't know any of the other N - K people's gender (some of whom will be male, some female)" – CHP Oct 16 '18 at 21:53
• Even though the sampling from B is random, there's some chance that the statistics won't reflect B. And B itself has some uncertainty when comparing it to A even if you had all members of B. So in essence I'm trying to understand how these combine - if they do, to determine my confidence. – CHP Oct 16 '18 at 21:59