Probability that an observation comes from population A or B?

Question

I'm a web developer looking into some basic statistics -- pardon me if I am using the wrong jargon. :)

Considering that:

• I have 2 populations (A and B; each have about 10,000 observations)
• For each population I know the size, the mean and the standard deviation
• I have one "lost" observation
• I know for sure that this observation belongs either in population A or in population B. It cannot co-exist in both populations.

Is it possible to determine the probabilities that the observation belongs in population A or population B? I would then compare each probability to determine which case is more likely.

I would prefer not to make an assumption about the distribution of each population. However, if necessary, it would be fair to assume that the populations are normally distributed.

In case it helps, I have some sample data available:

• observation x of interest: 0.85
• population A mean: 0.49832024649637213001, n: 10061, standard deviation: 0.26712151244680104078
• population B mean: 0.49646091156051916692, n: 9939, standard deviation: 0.26807810534781098689

What is the chance for observation x to be in population A? What is the chance for observation x to be in population B?

Note that I realize that the means and standard deviations of both populations are very similar. I don't mind if the probabilities are very similar too. This is just an example observation.

Answer 1

Let $X$ be the random variable for your observation, and $x$ a particular value (say 0.413). We can then use Bayes' Rule:

\begin{align} p(X \in A \mid X = x) &= \frac{p(X = x \mid X \in A) \, p(X \in A)}{p(X = x)} \\&= \frac{p(X = x \mid X \in A) \, p(X \in A)}{p(X = x \mid X \in A) \, p(X \in A) + p(X = x \mid X \in B) p(X \in B)} .\end{align}

Now the question becomes how to estimate each of these terms.

A reasonable thing to say for $p(X \in A) = 1 - p(X \in B)$ might be $\frac{n_A}{n_A + n_B}$, but it's going to depend on your assumpitons.

If you're willing to assume $A$ and $B$ are normal, then you could get $p(X = x \mid X \in A)$ and $p(X = x \mid X \in B)$ by the density of a normal distribution with the given mean and variance. This is going to give you almost $0.5$ probability for any possible $x$, I think, though I haven't plugged the numbers in. I'd also be very careful about even trusting that $A$ is more likely than $B$ if you get out say $0.502$; remember that the parameters you've given are estimates, and a little bit of randomness might move you one way or the other.

If you don't want to assume that $A$ and $B$ are normal, you could do some kind of density estimation, perhaps kernel density estimation, to estimate the probabilities. If the populations are as similar as they appear from the means and variances — which you should definitely plot! — then again be very careful with interpreting this.

This is essentially a naive Bayes classifier, though here because your data is one-dimensional the "naive" assumption is unnecessary.

Answer 2

What you're describing is a mixture model of which the most famous example is a the Gaussian Mixture Model.

Mixture models have $k$ sub-populations each with their own to distribution. To sample a single observation from the mixture distribution, you first roll a die to determine which of the $k$ sub-populations the observation will be drawn from, say $i$. Then you look up on a table the parameters (such as mean and variance) of $i$-th sub-population; finally you sample the final observation from the distribution for the $i$-th population, e.g. $\mathcal{N}(\mu_i, \sigma^2_i)$. You don't have to use a normal distribution - the key point is the two-step process.

Given a single observation drawn from a known mixture model, you can determine which of the $k$ sub-populations it was most likely drawn from using Bayes theorem. Let's say our observation is $x$ and $\phi_i$ is the probability of the observation being drawn from the $i$-th sub-population.

We have

$$P(x<X<x+\epsilon|i) = \text{pdf}(x; \mu_i, \sigma_i^2)$$

and

$$P(i) = \phi_i$$

therefore

$$P(x<X<x+\epsilon) = \phi_i \; \text{pdf}(x; \mu_i, \sigma_i^2)$$

gives us the probability that the observation $x$ was drawn from the $i$-th sub-population; the largest probability over all $k$ sub-populations is the one it most likely came from.

Answer 3

Is it possible to determine the probabilities that the observation belongs in population A or population B? I would then compare each probability to determine which case is more likely.

The basic idea would be to compare how many members of Population A with value .85 with members of Population B with value .85. Assuming that the process that generated the lost data point is the same as the one that generated your other data, you can look at how many observations of Population A have value .85, versus Population B. However, you'll have to decide how wide of a window around .85 to take; you probably won't have any data points that are exactly .85000000. A narrow window will have a smaller sample size, but a larger window will be affected by data further and further away from .85. This is the variance/bias tradeoff.

Instead of looking at how many data points there actually are at .85, you can also look at how many there "should" be. For this, you need to make an assumption about the type of distribution, and estimate the parameters. For instance, if you assume that the distribution is normal, then you can use the mean and standard deviation to calculated the probability of .85. However, you should look at your data and see whether the distribution is actually normal.

If you calculate how many observations "should" be at .85 for A vs. B, this will give a Bayesian calculation of the probability that it is from A vs. B. However, to be truly rigorous, you have to do a meta-Bayesian analysis that, given your prior beliefs about the distributions, finds the probability distribution of the observed sample statistics of mean and standard deviation. If the standard deviation is .268, then the two-tailed p-value for the sample means differing by .002 is about 60%. That is, the probability of seeing as large, or larger, difference in the means purely by chance is about 60%. Thus, this data does not give a solid basis for thinking that the true means for A and B are actually different. Also, keep in mind that sample statistics generally have roughly half the significant figures as the sample size; the sample size of 9939 has four significant figures, so the statistics have only two. Twenty digits of precision is definitely overkill. The two-tailed p-value for getting a sample mean of .49646, given an actual mean of .5 and std of .268, is about 20%, which is still far from statistically significant. So the data is consistent with both means actually being .5.

Answer 4

Perhaps the question is a simpler one than assumed. If the means and standard deviations are from populations, couldn't we calculate the z-score for the observation? Doing so with regard to Population A results in a z of 1.316553468 and a quick check of z-table indicates that the proportion of values expected to be 0.85 or greater is 0.093994 (or about 9.40%) With regard to population B we end up with a z of 1.318791357 suggesting the proportion of values expected to be 0.85 or larger is 0.093619 (or about 9.36%). This would suggest that the observation is very slightly more representative or Population A than Population B.