Probability that the sample comes from a certain distribution

Question

Assume we have a data sample: $x_{1}, \dots, x_{n}$ from $n$ i.i.d. continuous random variables. Then, for simplicity, let us consider two distributions, $f(x)$ and $g(x)$. Is there any statistical way to decide, which distribution, $f(x)$ or $g(x)$, the sample rather belongs to? Would it make sense and would it be somehow consistent to compare the likelihoods $$ L_{1} = f(x_{1})f(x_{2})\cdots f(x_{n}) $$ and $$ L_{2} = g(x_{1})g(x_{2})\cdots g(x_{n})? $$ I am aware of tests (for example, KS test), but the tests reject at a given confidence level the hypothesis that the sample comes from a particular distribution. My goal is to quantify how likely is it that the sample is from a certain distribution.

Answer 1

phibog's goal is to find out the probability that a set of data samples belongs to the distribution with pdf $f(x)$. It can be expressed as $P(f(x)|x_1,x_2,...,x_n)$. From Bayes theorem,

$P(f(x)|x_1,x_2,...,x_n)=\frac{P(x_1,x_2,...,x_n|f(x))P(f(x))}{P(x_1,x_2,...,x_n|f(x))P(f(x))+P(x_1,x_2,...,x_n|g(x))P(g(x))},$

where $P(f(x))$ and $P(g(x))$ are a prior probabilities that the data samples belong to $f(x)$ and $g(x)$ respectively. Since no information is available about which distribution is more likely than another, simply let $P(f(x))=P(g(x))$. The above expression then becomes

$P(f(x)|x_1,x_2,...,x_n)=\frac{P(x_1,x_2,...,x_n|f(x))}{P(x_1,x_2,...,x_n|f(x))+P(x_1,x_2,...,x_n|g(x))}$.

When $x_1,x_2,...,x_n$ are all independent, finally we have

$P(f(x)|x_1,x_2,...,x_n)=\frac{L_1}{L_1+L_2}$,

where $L_1$ and $L_2$ have been defined in the OP.

Answer 2

A simple method here is to use Bayesian analysis. For notational simplicity, let's "parameterise" your family of (two) distributions by defining the probability density $h$ by:

$$h(x|\theta) = \begin{cases} f(x) & & & \text{if } \theta = 0, \\[6pt] g(x) & & & \text{if } \theta = 1. \\[6pt] \end{cases}$$

The binary parameter $\theta$ now tells you which is the true distribution --- $f$ or $g$. Now, to determine which is the correct distribution given your data, we set a prior probability $\pi = \mathbb{P}(\theta = 0)$ that the data come from the distribution $f$, and we compute the corresponding posterior probability:

$$\mathbb{P}(\theta=0 | \mathbf{x}) = \frac{ \pi \prod_{i=1}^n f(x_i)}{ \pi \prod_{i=1}^n f(x_i) + (1-\pi) \prod_{i=1}^n g(x_i)}.$$

This tells you the posterior probability that the data come from the distribution $f$ (and the corresponding probability that they come from $g$ is one minus this probability).

Answer 3

If one of your distribution is a more/less restricted version of the other, then the Likelihood-ratio test might be what you want. More specifically:

Assuming $H_0$ is true, there is a fundamental result by Samuel S. Wilks: As the sample size $n$ approaches $\infty$, the test statistic $-2\log(\lambda)$ asymptotically will be chi-squared distributed ($\chi ^{2}$) with degrees of freedom equal to the difference in dimensionality of $\Theta$ and $\Theta_0$.

where $\lambda$ is the likelihood ratio $\frac{L_\mathrm{null}}{L_\mathrm{alter}}$.

I am not sure if this method would be appropriate for comparing likelihoods from two arbitrary distributions though.

Answer 4

The data either come from the specified model or they do not. There is no probability regarding this concern. The candidate model serves as a convenient yet imperfect representation of the data generative process. To compare how competing models fit your data you can use a criterion such as AIC based on the likelihood. You can base your choice of probability model at least in part on this criterion. Regarding the K-S test, rather than defining an "acceptance/rejection" region you could view the p-value as the weight of the evidence regarding a particular model. For various competing models you could compare K-S p-values analogous to comparing AIC values. Models with larger p-values do a better job representing your data.

You may instead be interested in a Bayesian belief probability regarding whether a candidate model is the "right" model. This would be the realm of Bayes factor or other type of model selection and would require a prior distribution on the candidate models. According to the adage, "All models are wrong but some are useful," you could end up assigning a 0% probability to every candidate model. You are also free to assign any other prior distribution. This would allow you to arrive at any conclusion for the belief probability that the sample comes from a certain distribution.

Answer 5

To answer your question, you can select from a few available tests. To quote Wikipedia on the Kolmogorov–Smirnov test where other competitive tests are also cited (placed in bold):

Kolmogorov–Smirnov test (K–S test or KS test) is a nonparametric test of the equality of continuous (or discontinuous, see Section 2.2), one-dimensional probability distributions that can be used to compare a sample with a reference probability distribution (one-sample K–S test), or to compare two samples (two-sample K–S test)...the Kolmogorov–Smirnov statistic quantifies a distance between the empirical distribution function of the sample and the cumulative distribution function of the reference distribution, or between the empirical distribution functions of two samples. The null distribution of this statistic is calculated under the null hypothesis that the sample is drawn from the reference distribution (in the one-sample case) or that the samples are drawn from the same distribution (in the two-sample case). In the one-sample case, the distribution considered under the null hypothesis may be continuous (see Section 2), purely discrete or mixed (see Section 2.2). In the two-sample case (see Section 3), the distribution considered under the null hypothesis is a continuous distribution but is otherwise unrestricted. The two-sample K–S test is one of the most useful and general nonparametric methods for comparing two samples, as it is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples. The Kolmogorov–Smirnov test can be modified to serve as a goodness of fit test. In the special case of testing for normality of the distribution, samples are standardized and compared with a standard normal distribution. This is equivalent to setting the mean and variance of the reference distribution equal to the sample estimates, and it is known that using these to define the specific reference distribution changes the null distribution of the test statistic (see Test with estimated parameters). Various studies have found that, even in this corrected form, the test is less powerful for testing normality than the Shapiro–Wilk test or Anderson–Darling test.[1] However, these other tests have their own disadvantages. For instance the Shapiro–Wilk test is known not to work well in samples with many identical values.

I trust this is a good start.

[EDIT] To answer the question as to whether one could develop an implied probability of actually being a member of a specified distribution, likely difficult in practice. To confirm my opinion, one could first compute the statistic that is the basis for say the KS test for say 10,000 pairs where both distributions are random draws from a selected single parent distribution. Tabulate the empirical distribution for the KS test statistic. Likely, accuracy of the developed probabilities would be sensitive to the test chosen, simulation run length, and the particular parent distribution. Note: this is only a Type I Error assessment. One could postulate an appropriate different distribution of concern for the second distribution to assess a Type II Error.