# Probability that the sample comes from a certain distribution

Assume we have a data sample: $$x_{1}, \dots, x_{n}$$ from $$n$$ i.i.d. continuous random variables. Then, for simplicity, let us consider two distributions, $$f(x)$$ and $$g(x)$$. Is there any statistical way to decide, which distribution, $$f(x)$$ or $$g(x)$$, the sample rather belongs to? Would it make sense and would it be somehow consistent to compare the likelihoods $$L_{1} = f(x_{1})f(x_{2})\cdots f(x_{n})$$ and $$L_{2} = g(x_{1})g(x_{2})\cdots g(x_{n})?$$ I am aware of tests (for example, KS test), but the tests reject at a given confidence level the hypothesis that the sample comes from a particular distribution. My goal is to quantify how likely is it that the sample is from a certain distribution.

• How about seeing which one has higher probability density at the observed points? Oct 7 '20 at 13:31
• Well, this is what I proposed, but likelihood is not probability. Oct 7 '20 at 13:36
• The decision requires adopting a prior probability for the two distributions. Bayes' Theorem asserts that the prior odds, multiplied by the likelihood ratio, gives the posterior odds. You can use those to make this decision. Without the prior odds, all you have is the likelihood ratio and you are correct that is not a probability.
– whuber
Oct 7 '20 at 14:40

phibog's goal is to find out the probability that a set of data samples belongs to the distribution with pdf $$f(x)$$. It can be expressed as $$P(f(x)|x_1,x_2,...,x_n)$$. From Bayes theorem,

$$P(f(x)|x_1,x_2,...,x_n)=\frac{P(x_1,x_2,...,x_n|f(x))P(f(x))}{P(x_1,x_2,...,x_n|f(x))P(f(x))+P(x_1,x_2,...,x_n|g(x))P(g(x))},$$

where $$P(f(x))$$ and $$P(g(x))$$ are a prior probabilities that the data samples belong to $$f(x)$$ and $$g(x)$$ respectively. Since no information is available about which distribution is more likely than another, simply let $$P(f(x))=P(g(x))$$. The above expression then becomes

$$P(f(x)|x_1,x_2,...,x_n)=\frac{P(x_1,x_2,...,x_n|f(x))}{P(x_1,x_2,...,x_n|f(x))+P(x_1,x_2,...,x_n|g(x))}$$.

When $$x_1,x_2,...,x_n$$ are all independent, finally we have

$$P(f(x)|x_1,x_2,...,x_n)=\frac{L_1}{L_1+L_2}$$,

where $$L_1$$ and $$L_2$$ have been defined in the OP.

If one of your distribution is a more/less restricted version of the other, then the Likelihood-ratio test might be what you want. More specifically:

Assuming $$H_0$$ is true, there is a fundamental result by Samuel S. Wilks: As the sample size $$n$$ approaches $$\infty$$, the test statistic $$-2\log(\lambda)$$ asymptotically will be chi-squared distributed ($$\chi ^{2}$$) with degrees of freedom equal to the difference in dimensionality of $$\Theta$$ and $$\Theta_0$$.

where $$\lambda$$ is the likelihood ratio $$\frac{L_\mathrm{null}}{L_\mathrm{alter}}$$.

I am not sure if this method would be appropriate for comparing likelihoods from two arbitrary distributions though.

To answer your question, you can select from a few available tests. To quote Wikipedia on the Kolmogorov–Smirnov test where other competitive tests are also cited (placed in bold):

Kolmogorov–Smirnov test (K–S test or KS test) is a nonparametric test of the equality of continuous (or discontinuous, see Section 2.2), one-dimensional probability distributions that can be used to compare a sample with a reference probability distribution (one-sample K–S test), or to compare two samples (two-sample K–S test)...the Kolmogorov–Smirnov statistic quantifies a distance between the empirical distribution function of the sample and the cumulative distribution function of the reference distribution, or between the empirical distribution functions of two samples. The null distribution of this statistic is calculated under the null hypothesis that the sample is drawn from the reference distribution (in the one-sample case) or that the samples are drawn from the same distribution (in the two-sample case). In the one-sample case, the distribution considered under the null hypothesis may be continuous (see Section 2), purely discrete or mixed (see Section 2.2). In the two-sample case (see Section 3), the distribution considered under the null hypothesis is a continuous distribution but is otherwise unrestricted. The two-sample K–S test is one of the most useful and general nonparametric methods for comparing two samples, as it is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples. The Kolmogorov–Smirnov test can be modified to serve as a goodness of fit test. In the special case of testing for normality of the distribution, samples are standardized and compared with a standard normal distribution. This is equivalent to setting the mean and variance of the reference distribution equal to the sample estimates, and it is known that using these to define the specific reference distribution changes the null distribution of the test statistic (see Test with estimated parameters). Various studies have found that, even in this corrected form, the test is less powerful for testing normality than the Shapiro–Wilk test or Anderson–Darling test. However, these other tests have their own disadvantages. For instance the Shapiro–Wilk test is known not to work well in samples with many identical values.

I trust this is a good start.

[EDIT] To answer the question as to whether one could develop an implied probability of actually being a member of a specified distribution, likely difficult in practice. To confirm my opinion, one could first compute the statistic that is the basis for say the KS test for say 10,000 pairs where both distributions are random draws from a selected single parent distribution. Tabulate the empirical distribution for the KS test statistic. Likely, accuracy of the developed probabilities would be sensitive to the test chosen, simulation run length, and the particular parent distribution. Note: this is only a Type I Error assessment. One could postulate an appropriate different distribution of concern for the second distribution to assess a Type II Error.

• As I wrote in my question, I am aware of the tests. Tests do not give the probability, they reject the null hypothesis at a certain level of confidence. Oct 7 '20 at 13:46
• Added an Edit Section, that may be of assistance. Oct 7 '20 at 14:33
• @phibog The test statistic of KS test does follow a probability distribution (the Kolmogorov distribution) and I imagine one could develop some probabilistic interpretation over the results of the test; the p-value is "the probability of obtaining test results at least as extreme as the results actually observed" after all. Oct 7 '20 at 15:59