Imagine we're drawing samples from two normal distributions(N1 ~ (mu1, std1), N2 ~ (mu2, std2)) with different parameters.

How to calculate probability of specific sample being generated by first or the second process (the only two possible options)?

So, P(sample coming from N1) + P(sample coming form N2) = 1

P(sample coming from N1) = ?

  • $\begingroup$ Can you please clarify if you are trying to infer the proportion of samples N1 generate, or given a sample (presumably its value) the probability it would have come from N1? The text and mathematical formulation is currently giving conflicting signals. $\endgroup$ – B.Liu Apr 12 '19 at 15:49
  • $\begingroup$ Sure. We have access to only one sample. We know that it could've been generated by one of two distributions. What is probability that it is generetad by the first one? That is the problem. $\endgroup$ – agawa Apr 12 '19 at 15:53
  • $\begingroup$ @ArtemMavrin If we look at both distributions as one mixture model, I don't see how we can infer which distribution produced the sample $\endgroup$ – agawa Apr 12 '19 at 15:55
  • $\begingroup$ Do you know the proportion of N1-generated sample to that of N2-generated samples in general, i.e. if N1 generates say 60% of the samples and N2 generates 40% of them? If so using the Baye's Theorem on P(sample from N1 | sample has value x) should solve the problem. $\endgroup$ – B.Liu Apr 12 '19 at 16:01
  • $\begingroup$ If you like to think in terms of bayesian analysis, initial probability of sample being form N1 is 0.5. The thing is that if sample value is closer to the mean of N1 it more likely to be generated by N1 because N1 has more probability mass around that area than N2 which is further away (with its mean). $\endgroup$ – agawa Apr 12 '19 at 16:10

Assuming we know the probability that a sample generated from the process comes from N1 (or N2), this is a straightforward application of the Bayes' Theorem.

We first define:
$X$ as the random variable corresponding to the random sample the OP has,
$G_1$ as the event where $X$ is generated by N1, and
$G_2$ as the event where $X$ is generated by N2.

We are looking for $\mathbb{P}(G_1 | X = x)$, the probability where the sample observed is generated by N1.

By Bayes' Theorem, this leads to:

$$\mathbb{P}(G_1 | X = x) = \frac{f(X = x | G_1) \mathbb{P}(G_1)}{f(X = x)}$$

Using law of total probability, we expand the denominator to obtain:

$$\mathbb{P}(G_1 | X = x) = \frac{f(X = x | G_1) \mathbb{P}(G_1)}{f(X = x | G_1) \mathbb{P}(G_1) + f(X = x | G_2) \mathbb{P}(G_2)}$$

$\mathbb{P}(G_1)$ is the probability that a sample generated from the process comes from N1, and $f(X = x | G_1)$ is the probability density over support of $X$ assuming N1 generates all the samples, same applies to quantities related to $G_2$.

One can then obtain an analytical formula by substituting the right probability values and probability density functions.

  • $\begingroup$ Generally makes sense, but I'm confused on how to handle probability density functions of continuous distributions. If priors P(G1) and P(G2) are 0.5, we can simplify to: P(G1|X=x) = f(X=x|G1) / (f(X=x|G1) + f(X=x|G2)). How do I get exact probability by dividing PDFs. Do I integrate? Over what range? Or does 𝑓(𝑋=𝑥|𝐺1) means probability of sampled X being EXACTLY x which is 0 for continuous distributions. $\endgroup$ – agawa Apr 12 '19 at 20:44
  • 1
    $\begingroup$ I think i got it now, thanks. I don't need probability, I can multiply probability density and probability of discrete event. $\endgroup$ – agawa Apr 12 '19 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.