Finding the probability that a set of measurement belongs to a set of normal distributions

Question

Suppose that we are given a set of $n \cdot k $ normal distributions so that a given measurement of $k$ values either comes from a given set of $k$ normal distributions or not. How do we calculate the odds that it comes from each of the $k$-sets of normal distributions? To clarify (although I'm mostly interested in the general method), take an example: let $k=5$ and $n=3$:

   k/n {mean,stdv} |    1    |    2    |    3    |
         1            {20,5}    {21,6}    {24,3} 
         2            {31,4}    {28,5}    {28,4}
         3            {15,9}    {18,4}    {17,4} 
         4            {10,4}    {8,2}     {6,1}
         5            {6,2}     {5,4}     {6,3}

Now how to we calculate which of the above sets the measurement $\{21,30,13,12,4\}$ is most likely to come from, given that it comes from one of them? As in, what is the general method to calculate this? (Intuitively, we see that it is likely to come from the first $k$-set, as each number in the measurement falls within one standard deviation of the values in that $k$-set).

Thanks for any input.

Answer 1

You can use something like linear discriminant analysis. However, in your case we do not have equal covariances matrices, as it is assumed in linear discriminant analysis. Let's say that we have stochastic variables $X$ and $G$ where $X$ signifies the measurement, and $G$ the underlying distribution (thus, $G$ takes values in $\{1,2,3\}$ in the example). The coordinates of $X$ are independent and furthermore, $X$ and $G$ are independent. Basically, we're interested in the quantity

$$ P(G = k | X =x ), $$

where $x$ is a realization of $X$ and $k\in \{1,2,3\}$. For clarity, I'll just continue the example. The generalization is straight-forward.

By Bayes' theorem, we get that

$$ P(G = k | X =x ) = \dfrac{f_X(x|G=k)P(G = k)}{f_X(x)}. $$

In your example, we have no prior information on the distributions, so let's assume that they are equally likely before we observe $x$, i.e. $P(G = k) = 1/3$ for $k \in \{1,2,3\}$. We can rewrite the denominator using our assumptions and using that the distribution of $X$ is a mixture of distributions.

$$ f_X(x) = \sum_{i=1}^3 f_X ( x | G = i) P(G = i) $$

Thus, we get that

$$ P(G= k |X=x) = \dfrac{f_X ( x | G = k)}{\sum_{i=1}^3 f_X ( x | G = i)}. $$

These will be numbers in $(0,1)$ and they sum to $1$, thus it is tempting to see them as probabilities. In a frequentist set-up, let's consider how to interpret these probabilities. There is a true distribution. Hence, this distribution has probability 1 of being the true distribution no matter the outcome. Of course, we do not know which one it is, though. This is analogous to the interpretation of confidence intervals in a parametric model. In a frequentist interpretation, there is a true parameter value. When we observe data and construct a 95% confidence interval, this interval either contains the true value or it doesn't. However, when we repeat the experiment infinitely many times, we would in 95% of the cases get an interval containing the true parameter.

I would say that, if one were to interpret the above probabilities in a frequentist manner, one would have to interpret them in terms of repetitions of the experiment, all giving $x$ as the outcome. A Bayesian interpretation is perhaps more straight-forward, as degrees of belief in the different states. In this case the probabilities gives us the posterior distribution.