# How to estimate which distribution a new observation belongs to, when the distributions are given by a set of obervations

The question is that I have several probability distributions. For each distribution $P_i$, I don't know what it is. Instead, I have a set of observations $\{x_i^j\}_{j=1}^{n_i}$ drawn from each of the $P_i$. Now for a new observation $x_t$, is there a way to estimate the likelihood of each $P_i$ the new observation belongs to? Or which $P_i$ it most likely belongs to? Here $x_i^j, x_t \in R^m$.

I do hope for a method that can directly calculate the likelihoods without making assumptions on the distributions and calculating their parameters first. But honestly I don't know if such methods even exist.

Thanks!

• Have you looked at classification algorithms. For example k nearest neighbors. Jul 3, 2015 at 7:52
• @shane Thanks! I didn't even realize that it was a classification problem! In fact the observations $\{x_i^j\}_{j=1}^{n_i}$ are labeled by $i$, so it is a supervised learning problem. Jul 3, 2015 at 15:57

In fact (if I do understand your problem), you can easily construct a trivial example which has an infinite number of non-unique sets of $P_i$ from which the $x_i$ could have come from.
For example: suppose you have two hidden discrete RVs such that $P_1 = 1$ with probability 1 and $P_2 = 2$ with probabilty 1. Assume also, that the probability you draw an $x_i$ from either of the two distributions is $\frac{1}{2}$ and that these drawings are indept.
You could never distinguish this scenario from one where the distributions where defined such that: $P_1(X=1)=0.25$ and $P_1(X=2)=0.75$, whilst $P_2(X=1)=0.75$ and $P_2(X=2)=0.25$ and the $x_i$ were independently drawn from $P_1$ and $P_2$ with probability $\frac{1}{2}$.
• Hi Jonathan, thanks for your answer! Maybe I didn't state my question well, so sorry if I have confused you. @shane just reminded me that my question, in essence, is a classification problem. I have a set of observations $\{x_i^j\}_{j=1}^{n_i}$ drawn from each distribution (class) $P_i$, and I want to know the probability that a new observation belongs to each of the distribution (class). I now have realized it could be solved by training a classifier. Jul 3, 2015 at 15:55