3
$\begingroup$

I'm a web developer looking into some basic statistics -- pardon me if I am using the wrong jargon. :)

Considering that:

  • I have 2 populations (A and B; each have about 10,000 observations)
  • For each population I know the size, the mean and the standard deviation
  • I have one "lost" observation
  • I know for sure that this observation belongs either in population A or in population B. It cannot co-exist in both populations.

Is it possible to determine the probabilities that the observation belongs in population A or population B? I would then compare each probability to determine which case is more likely.

I would prefer not to make an assumption about the distribution of each population. However, if necessary, it would be fair to assume that the populations are normally distributed.

In case it helps, I have some sample data available:

  • observation x of interest: 0.85
  • population A mean: 0.49832024649637213001, n: 10061, standard deviation: 0.26712151244680104078
  • population B mean: 0.49646091156051916692, n: 9939, standard deviation: 0.26807810534781098689

What is the chance for observation x to be in population A? What is the chance for observation x to be in population B?

Note that I realize that the means and standard deviations of both populations are very similar. I don't mind if the probabilities are very similar too. This is just an example observation.

$\endgroup$
  • $\begingroup$ Without knowing more about the data or making distributional assumptions, then mean and sd alone do not tell us almost anything about the probabilities. So this is insufficient information to answer the question, especially since the characteristics of both groups are very close. $\endgroup$ – Tim Aug 20 '18 at 13:22
  • $\begingroup$ Given your description of the data, I assume you want to classify the data point b it's mean value. Given the essentially identical numerics of data set A and data set B, I think this is impossible. As far as I can tell, any standard statistical test would say the population A and population B are indistinguishable. $\endgroup$ – aginensky Aug 20 '18 at 15:06
  • $\begingroup$ @Tim if necessary it is fair to assume that the data is normally distributed. $\endgroup$ – Tom Aug 20 '18 at 15:25
  • $\begingroup$ @aginensky please do not be discouraged by their similarity. I do not require statistical significance. Having some kind of difference in probability is better than having nothing at all. :) $\endgroup$ – Tom Aug 20 '18 at 15:26
3
$\begingroup$

Let $X$ be the random variable for your observation, and $x$ a particular value (say 0.413). We can then use Bayes' Rule:

\begin{align} p(X \in A \mid X = x) &= \frac{p(X = x \mid X \in A) \, p(X \in A)}{p(X = x)} \\&= \frac{p(X = x \mid X \in A) \, p(X \in A)}{p(X = x \mid X \in A) \, p(X \in A) + p(X = x \mid X \in B) p(X \in B)} .\end{align}

Now the question becomes how to estimate each of these terms.

A reasonable thing to say for $p(X \in A) = 1 - p(X \in B)$ might be $\frac{n_A}{n_A + n_B}$, but it's going to depend on your assumpitons.

If you're willing to assume $A$ and $B$ are normal, then you could get $p(X = x \mid X \in A)$ and $p(X = x \mid X \in B)$ by the density of a normal distribution with the given mean and variance. This is going to give you almost $0.5$ probability for any possible $x$, I think, though I haven't plugged the numbers in. I'd also be very careful about even trusting that $A$ is more likely than $B$ if you get out say $0.502$; remember that the parameters you've given are estimates, and a little bit of randomness might move you one way or the other.

If you don't want to assume that $A$ and $B$ are normal, you could do some kind of density estimation, perhaps kernel density estimation, to estimate the probabilities. If the populations are as similar as they appear from the means and variances — which you should definitely plot! — then again be very careful with interpreting this.

This is essentially a naive Bayes classifier, though here because your data is one-dimensional the "naive" assumption is unnecessary.

$\endgroup$
  • $\begingroup$ Excellent. Sounds like the example "Sex classification" of your link to to the naive Bayes classifier is very relevant? I'll try to apply it to my data. $\endgroup$ – Tom Aug 20 '18 at 16:05
1
$\begingroup$

What you're describing is a mixture model of which the most famous example is a the Gaussian Mixture Model.

Mixture models have $k$ sub-populations each with their own to distribution. To sample a single observation from the mixture distribution, you first roll a die to determine which of the $k$ sub-populations the observation will be drawn from, say $i$. Then you look up on a table the parameters (such as mean and variance) of $i$-th sub-population; finally you sample the final observation from the distribution for the $i$-th population, e.g. $\mathcal{N}(\mu_i, \sigma^2_i)$. You don't have to use a normal distribution - the key point is the two-step process.

Given a single observation drawn from a known mixture model, you can determine which of the $k$ sub-populations it was most likely drawn from using Bayes theorem. Let's say our observation is $x$ and $\phi_i$ is the probability of the observation being drawn from the $i$-th sub-population.

We have

$$P(x<X<x+\epsilon|i) = \text{pdf}(x; \mu_i, \sigma_i^2)$$

and

$$P(i) = \phi_i$$

therefore

$$P(x<X<x+\epsilon) = \phi_i \; \text{pdf}(x; \mu_i, \sigma_i^2)$$

gives us the probability that the observation $x$ was drawn from the $i$-th sub-population; the largest probability over all $k$ sub-populations is the one it most likely came from.

$\endgroup$
  • $\begingroup$ This sounds great. However, perhaps I am missing the obvious, but wouldn't I still need to know how to compute the probability density function? $\endgroup$ – Tom Aug 20 '18 at 15:37
  • 1
    $\begingroup$ Yes. And you do have to have an opinion on the distribution to do the calculation. But in your case you know the mean and variance so if you're willing to assume it's normal, you can use the pdf of the normal distribution, which certainly will be built into whatever software you're using e.g. $dnorm()$. $\endgroup$ – olooney Aug 20 '18 at 16:05
1
$\begingroup$

Is it possible to determine the probabilities that the observation belongs in population A or population B? I would then compare each probability to determine which case is more likely.

The basic idea would be to compare how many members of Population A with value .85 with members of Population B with value .85. Assuming that the process that generated the lost data point is the same as the one that generated your other data, you can look at how many observations of Population A have value .85, versus Population B. However, you'll have to decide how wide of a window around .85 to take; you probably won't have any data points that are exactly .85000000. A narrow window will have a smaller sample size, but a larger window will be affected by data further and further away from .85. This is the variance/bias tradeoff.

Instead of looking at how many data points there actually are at .85, you can also look at how many there "should" be. For this, you need to make an assumption about the type of distribution, and estimate the parameters. For instance, if you assume that the distribution is normal, then you can use the mean and standard deviation to calculated the probability of .85. However, you should look at your data and see whether the distribution is actually normal.

If you calculate how many observations "should" be at .85 for A vs. B, this will give a Bayesian calculation of the probability that it is from A vs. B. However, to be truly rigorous, you have to do a meta-Bayesian analysis that, given your prior beliefs about the distributions, finds the probability distribution of the observed sample statistics of mean and standard deviation. If the standard deviation is .268, then the two-tailed p-value for the sample means differing by .002 is about 60%. That is, the probability of seeing as large, or larger, difference in the means purely by chance is about 60%. Thus, this data does not give a solid basis for thinking that the true means for A and B are actually different. Also, keep in mind that sample statistics generally have roughly half the significant figures as the sample size; the sample size of 9939 has four significant figures, so the statistics have only two. Twenty digits of precision is definitely overkill. The two-tailed p-value for getting a sample mean of .49646, given an actual mean of .5 and std of .268, is about 20%, which is still far from statistically significant. So the data is consistent with both means actually being .5.

$\endgroup$
  • $\begingroup$ I really like your first suggestion of just binning the data around 0.85 and seeing how often it occurs in each population! It's pragmatic and simple to implement. I'm curious though, if the data is highly skewed, the bin sizes may have to be different at different values, right? $\endgroup$ – Tom Aug 20 '18 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.