# Probability that the sample comes from a certain distribution

Assume we have a data sample: $$x_{1}, \dots, x_{n}$$ from $$n$$ i.i.d. continuous random variables. Then, for simplicity, let us consider two distributions, $$f(x)$$ and $$g(x)$$. Is there any statistical way to decide, which distribution, $$f(x)$$ or $$g(x)$$, the sample rather belongs to? Would it make sense and would it be somehow consistent to compare the likelihoods $$L_{1} = f(x_{1})f(x_{2})\cdots f(x_{n})$$ and $$L_{2} = g(x_{1})g(x_{2})\cdots g(x_{n})?$$ I am aware of tests (for example, KS test), but the tests reject at a given confidence level the hypothesis that the sample comes from a particular distribution. My goal is to quantify how likely is it that the sample is from a certain distribution.

• How about seeing which one has higher probability density at the observed points? Oct 7, 2020 at 13:31
• Well, this is what I proposed, but likelihood is not probability. Oct 7, 2020 at 13:36
• The decision requires adopting a prior probability for the two distributions. Bayes' Theorem asserts that the prior odds, multiplied by the likelihood ratio, gives the posterior odds. You can use those to make this decision. Without the prior odds, all you have is the likelihood ratio and you are correct that is not a probability.
– whuber
Oct 7, 2020 at 14:40
• @whuber, I am familiar with Bayes' theorem $P(A|B)=\frac{P(B|A)P(A)}{P(B)}$ when $P(\cdot)$ stands for a probability and $P(B)>0$, but does a similar equation necessarily hold for $L_1$ and $L_2$, which are not probabilities? Is that rigorous, or a heuristic?
– Joe
Jan 11, 2022 at 21:09
• @Joe I see. Although that is not rigorous, it can be made rigorous in special (but common) cases with a suitable limiting argument or with measure theory. Intuitively, you need for there to exist a version of the conditional density that is continuous in a neighborhood of the data.
– whuber
Jan 12, 2022 at 16:36

phibog's goal is to find out the probability that a set of data samples belongs to the distribution with pdf $$f(x)$$. It can be expressed as $$P(f(x)|x_1,x_2,...,x_n)$$. From Bayes theorem,

$$P(f(x)|x_1,x_2,...,x_n)=\frac{P(x_1,x_2,...,x_n|f(x))P(f(x))}{P(x_1,x_2,...,x_n|f(x))P(f(x))+P(x_1,x_2,...,x_n|g(x))P(g(x))},$$

where $$P(f(x))$$ and $$P(g(x))$$ are a prior probabilities that the data samples belong to $$f(x)$$ and $$g(x)$$ respectively. Since no information is available about which distribution is more likely than another, simply let $$P(f(x))=P(g(x))$$. The above expression then becomes

$$P(f(x)|x_1,x_2,...,x_n)=\frac{P(x_1,x_2,...,x_n|f(x))}{P(x_1,x_2,...,x_n|f(x))+P(x_1,x_2,...,x_n|g(x))}$$.

When $$x_1,x_2,...,x_n$$ are all independent, finally we have

$$P(f(x)|x_1,x_2,...,x_n)=\frac{L_1}{L_1+L_2}$$,

where $$L_1$$ and $$L_2$$ have been defined in the OP.

• Since the random variables are continuous, aren't the right hand sides of your second and third equations each a fraction with zero in both the numerator and denominator?
– Joe
Jan 11, 2022 at 20:48

A simple method here is to use Bayesian analysis. For notational simplicity, let's "parameterise" your family of (two) distributions by defining the probability density $$h$$ by:

$$h(x|\theta) = \begin{cases} f(x) & & & \text{if } \theta = 0, \\[6pt] g(x) & & & \text{if } \theta = 1. \\[6pt] \end{cases}$$

The binary parameter $$\theta$$ now tells you which is the true distribution --- $$f$$ or $$g$$. Now, to determine which is the correct distribution given your data, we set a prior probability $$\pi = \mathbb{P}(\theta = 0)$$ that the data come from the distribution $$f$$, and we compute the corresponding posterior probability:

$$\mathbb{P}(\theta=0 | \mathbf{x}) = \frac{ \pi \prod_{i=1}^n f(x_i)}{ \pi \prod_{i=1}^n f(x_i) + (1-\pi) \prod_{i=1}^n g(x_i)}.$$

This tells you the posterior probability that the data come from the distribution $$f$$ (and the corresponding probability that they come from $$g$$ is one minus this probability).

• I understand this for discrete probabilities, but is it rigorous for PDF's of continuous r.v.? If we changed $f$ only at the point $x_1$ (a set of measure zero), wouldn't it still be a PDF for the same distribution, but give a different answer?
– Joe
Jan 11, 2022 at 20:51
• @Joe The discussion at stats.stackexchange.com/questions/248476 looks relevant.
– whuber
Jan 11, 2022 at 21:13
• @whuber, yes, thank you. So if I understood that post, we need to assume certain continuity conditions hold on $f$ and $g$
– Joe
Jan 12, 2022 at 16:34
• @Joe Yes, that's right.
– whuber
Jan 12, 2022 at 16:37
• @Joe: If you have a look at that discussion, you will see that we don't necessarily need to impose continuity on the densities. But we do need to ensure that $f$ and $g$ are densities defined with respect to the same dominating measure. So, for example, if we arbitrarily decide to triple the value $f(x_i)$ (i.e., at the particular point $x_i$) (so it is still a density, but no longer continuous) then we would also have to triple the value $g(x_i)$. Using the above formula for the posterior you should be able to verify that this change will not affect the posterior probability at issue.
– Ben
Jan 12, 2022 at 21:26

If one of your distribution is a more/less restricted version of the other, then the Likelihood-ratio test might be what you want. More specifically:

Assuming $$H_0$$ is true, there is a fundamental result by Samuel S. Wilks: As the sample size $$n$$ approaches $$\infty$$, the test statistic $$-2\log(\lambda)$$ asymptotically will be chi-squared distributed ($$\chi ^{2}$$) with degrees of freedom equal to the difference in dimensionality of $$\Theta$$ and $$\Theta_0$$.

where $$\lambda$$ is the likelihood ratio $$\frac{L_\mathrm{null}}{L_\mathrm{alter}}$$.

I am not sure if this method would be appropriate for comparing likelihoods from two arbitrary distributions though.

The data either come from the specified model or they do not. There is no probability regarding this concern. The candidate model serves as a convenient yet imperfect representation of the data generative process. To compare how competing models fit your data you can use a criterion such as AIC based on the likelihood. You can base your choice of probability model at least in part on this criterion. Regarding the K-S test, rather than defining an "acceptance/rejection" region you could view the p-value as the weight of the evidence regarding a particular model. For various competing models you could compare K-S p-values analogous to comparing AIC values. Models with larger p-values do a better job representing your data.

You may instead be interested in a Bayesian belief probability regarding whether a candidate model is the "right" model. This would be the realm of Bayes factor or other type of model selection and would require a prior distribution on the candidate models. According to the adage, "All models are wrong but some are useful," you could end up assigning a 0% probability to every candidate model. You are also free to assign any other prior distribution. This would allow you to arrive at any conclusion for the belief probability that the sample comes from a certain distribution.

To answer your question, you can select from a few available tests. To quote Wikipedia on the Kolmogorov–Smirnov test where other competitive tests are also cited (placed in bold):

Kolmogorov–Smirnov test (K–S test or KS test) is a nonparametric test of the equality of continuous (or discontinuous, see Section 2.2), one-dimensional probability distributions that can be used to compare a sample with a reference probability distribution (one-sample K–S test), or to compare two samples (two-sample K–S test)...the Kolmogorov–Smirnov statistic quantifies a distance between the empirical distribution function of the sample and the cumulative distribution function of the reference distribution, or between the empirical distribution functions of two samples. The null distribution of this statistic is calculated under the null hypothesis that the sample is drawn from the reference distribution (in the one-sample case) or that the samples are drawn from the same distribution (in the two-sample case). In the one-sample case, the distribution considered under the null hypothesis may be continuous (see Section 2), purely discrete or mixed (see Section 2.2). In the two-sample case (see Section 3), the distribution considered under the null hypothesis is a continuous distribution but is otherwise unrestricted. The two-sample K–S test is one of the most useful and general nonparametric methods for comparing two samples, as it is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples. The Kolmogorov–Smirnov test can be modified to serve as a goodness of fit test. In the special case of testing for normality of the distribution, samples are standardized and compared with a standard normal distribution. This is equivalent to setting the mean and variance of the reference distribution equal to the sample estimates, and it is known that using these to define the specific reference distribution changes the null distribution of the test statistic (see Test with estimated parameters). Various studies have found that, even in this corrected form, the test is less powerful for testing normality than the Shapiro–Wilk test or Anderson–Darling test. However, these other tests have their own disadvantages. For instance the Shapiro–Wilk test is known not to work well in samples with many identical values.

I trust this is a good start.

[EDIT] To answer the question as to whether one could develop an implied probability of actually being a member of a specified distribution, likely difficult in practice. To confirm my opinion, one could first compute the statistic that is the basis for say the KS test for say 10,000 pairs where both distributions are random draws from a selected single parent distribution. Tabulate the empirical distribution for the KS test statistic. Likely, accuracy of the developed probabilities would be sensitive to the test chosen, simulation run length, and the particular parent distribution. Note: this is only a Type I Error assessment. One could postulate an appropriate different distribution of concern for the second distribution to assess a Type II Error.

• As I wrote in my question, I am aware of the tests. Tests do not give the probability, they reject the null hypothesis at a certain level of confidence. Oct 7, 2020 at 13:46
• Added an Edit Section, that may be of assistance. Oct 7, 2020 at 14:33
• @phibog The test statistic of KS test does follow a probability distribution (the Kolmogorov distribution) and I imagine one could develop some probabilistic interpretation over the results of the test; the p-value is "the probability of obtaining test results at least as extreme as the results actually observed" after all. Oct 7, 2020 at 15:59