Suppose that we are given a set of $n \cdot k $ normal distributions so that a given measurement of $k$ values either comes from a given set of $k$ normal distributions or not. How do we calculate the odds that it comes from each of the $k$-sets of normal distributions? To clarify (although I'm mostly interested in the general method), take an example: let $k=5$ and $n=3$:

   k/n {mean,stdv} |    1    |    2    |    3    |
         1            {20,5}    {21,6}    {24,3} 
         2            {31,4}    {28,5}    {28,4}
         3            {15,9}    {18,4}    {17,4} 
         4            {10,4}    {8,2}     {6,1}
         5            {6,2}     {5,4}     {6,3}

Now how to we calculate which of the above sets the measurement $\{21,30,13,12,4\}$ is most likely to come from, given that it comes from one of them? As in, what is the general method to calculate this? (Intuitively, we see that it is likely to come from the first $k$-set, as each number in the measurement falls within one standard deviation of the values in that $k$-set).

Thanks for any input.

  • $\begingroup$ Can one assume independence between the measurements? $\endgroup$
    – swmo
    Commented Oct 11, 2014 at 12:24
  • $\begingroup$ Yes, we can :-) $\endgroup$
    – Sid
    Commented Oct 11, 2014 at 12:26
  • $\begingroup$ One idea would be to use values of the likelihoods (each three-dimensional in the example): pick the one with the highest likelihood value for the observed data. $\endgroup$
    – swmo
    Commented Oct 11, 2014 at 12:31
  • $\begingroup$ @swmo Can you clarify? I would like to calculate the probability for each $k$-set that the given measurement belongs to it, not just find the one with the highest probability. But maybe you are talking about the same thing? $\endgroup$
    – Sid
    Commented Oct 11, 2014 at 12:34
  • $\begingroup$ you can use linear discriminant analysis for this (or more precisely, this is what linear discriminant analysis is). $\endgroup$
    – swmo
    Commented Oct 11, 2014 at 12:48

1 Answer 1


You can use something like linear discriminant analysis. However, in your case we do not have equal covariances matrices, as it is assumed in linear discriminant analysis. Let's say that we have stochastic variables $X$ and $G$ where $X$ signifies the measurement, and $G$ the underlying distribution (thus, $G$ takes values in $\{1,2,3\}$ in the example). The coordinates of $X$ are independent and furthermore, $X$ and $G$ are independent. Basically, we're interested in the quantity

$$ P(G = k | X =x ), $$

where $x$ is a realization of $X$ and $k\in \{1,2,3\}$. For clarity, I'll just continue the example. The generalization is straight-forward.

By Bayes' theorem, we get that

$$ P(G = k | X =x ) = \dfrac{f_X(x|G=k)P(G = k)}{f_X(x)}. $$

In your example, we have no prior information on the distributions, so let's assume that they are equally likely before we observe $x$, i.e. $P(G = k) = 1/3$ for $k \in \{1,2,3\}$. We can rewrite the denominator using our assumptions and using that the distribution of $X$ is a mixture of distributions.

$$ f_X(x) = \sum_{i=1}^3 f_X ( x | G = i) P(G = i) $$

Thus, we get that

$$ P(G= k |X=x) = \dfrac{f_X ( x | G = k)}{\sum_{i=1}^3 f_X ( x | G = i)}. $$

These will be numbers in $(0,1)$ and they sum to $1$, thus it is tempting to see them as probabilities. In a frequentist set-up, let's consider how to interpret these probabilities. There is a true distribution. Hence, this distribution has probability 1 of being the true distribution no matter the outcome. Of course, we do not know which one it is, though. This is analogous to the interpretation of confidence intervals in a parametric model. In a frequentist interpretation, there is a true parameter value. When we observe data and construct a 95% confidence interval, this interval either contains the true value or it doesn't. However, when we repeat the experiment infinitely many times, we would in 95% of the cases get an interval containing the true parameter.

I would say that, if one were to interpret the above probabilities in a frequentist manner, one would have to interpret them in terms of repetitions of the experiment, all giving $x$ as the outcome. A Bayesian interpretation is perhaps more straight-forward, as degrees of belief in the different states. In this case the probabilities gives us the posterior distribution.

  • 1
    $\begingroup$ Thank you! I believe that for my project, it is satisfying enough to consider the value you get probabilities. I have only one follow-up question: how do we calculate $f_X ( x | G = k)$? $\endgroup$
    – Sid
    Commented Oct 11, 2014 at 14:41
  • $\begingroup$ If you know that $G = k$, then you know from what distribution $x$ is drawn. Thus, $f_X(x|G=k)$ will be equal to the density (taken in $x$) of the multidimensional normal distribution with parameters corresponding to the $k$'th underlying distribution. $\endgroup$
    – swmo
    Commented Oct 11, 2014 at 14:59
  • $\begingroup$ Are we talking about this function, multiplied $5$ times for each case? upload.wikimedia.org/math/7/3/a/… Or do I need to plug the values into this function: upload.wikimedia.org/math/7/1/a/… ? $\endgroup$
    – Sid
    Commented Oct 11, 2014 at 15:03
  • $\begingroup$ Both approaches will give the same thing as we have assumed independence: independence makes $\Sigma$ a diagonal matrix and the joint density becomes the product of the marginal ones. $\endgroup$
    – swmo
    Commented Oct 11, 2014 at 15:08
  • $\begingroup$ Gracias, you've been very helpful $\endgroup$
    – Sid
    Commented Oct 11, 2014 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.