How to get probability of sample coming from a distribution?

I have a set of observations drawn from an unknown distribution. Given a new observation $x$ I would like to ascertain the probability that $x$ was drawn from the same distribution.

My approach was to use kernel density estimation to estimate the pdf of the initial samples, and then use this to estimate the probability of $x$. It has now come to my attention that the $\mathrm{pdf}(x) \neq P(x)$.

How can I calculate $P(x)$?

• If you literally want such a probability (which is eminently reasonable) you need to supply additional information in the form of a prior distribution (over a set of possible distributions). Then Bayes' Theorem does the rest. There are other approaches to addressing needs similar to yours. For instance, a "non-parametric prediction interval" uses data drawn (independently) from a totally unknown distribution to erect an interval having a specified chance of including $x$. That doesn't involve computing $P(x)$, but perhaps it addresses your needs. Which approach do you want to take? – whuber Sep 17 '13 at 18:28
• Using the non-parametric prediction interval for $n$ data points, then if the new value is > the max or < the min I can say it has a 1-(n-1)/(n+1) probability of being part of the underlying distribution. But if it falls within the range, I cannot assign any confidence value to it. Essentially I am trying to have a confidence or value that a new data is an "outlier". Any help on this would be of great value – Aly Sep 17 '13 at 18:42
• You are quoting only one form of a nonparametric prediction interval. Although it's about the only useful form for small datasets (namely, from the min to the max), for larger datasets other ranges of order statistics provide finer control over the size of the interval. There are also prediction intervals for sets of iid future values, such as intervals for their maximum, their mean, and their median. All of these potentially are useful in your circumstances, especially if you need a formal hypothesis test. – whuber Sep 17 '13 at 19:57
• When you say 'sample' do you mean 'observation'? Sample usually refers to a set of observations. It's possible to read your question with sample carrying either meaning, but it changes the possible answers. – Glen_b Sep 17 '13 at 23:51
• @Glen_b Sorry I mean observation – Aly Sep 18 '13 at 9:48

You can extend your non-parametric method if your original sample is large enough.

Suppose you wanted to have a 95% probability of not rejecting the null hypothesis that your new observation comes from the same distribution if it in fact does: your critical region could be to reject the null hypothesis if your new observation is in the top $k$ or bottom $k$ of the now $n+1$ values.

So you want $\frac{2k}{n+1} = 1 - 0.95$, i.e. $k=0.05\frac{n+1}{2}$, giving the following pairs of values of $k$ and $n$

k   n
1   39
2   79
3  119
4  159

and the pattern is obvious. In reality, $n$ will be decided for you so you need to make a sensible choice in the circumstances.

In my view, you should choose $k$ (or, in general, the critical region) based on $n$ (or the original sample) before you look at the new observation, rather than looking at the full data and then trying to derive a probability from the new observation as seen: if the null hypothesis is in fact true then each position is equally likely, and the probability of being as extreme or more extreme than the new observation would be basing your conclusion on things that were not observed.

If you had some idea about the unknown original distribution or the possible alternative distribution, you could probably do better than this.

You can't get $P(x)$ for a continuous distribution. It is 0. To get $P(a \leq x \leq b)$ you would compute the following:

$$\int_{a}^{b} f(x) \ dx$$

• Is there a practical way to approach the problem I described above? One way that is common is to take some distance measure to the mean, but this data may be multi-modal and thus this distance may not mean much – Aly Sep 17 '13 at 18:26