# How to rigorously test if variables are drawn from a certain distribution

Say I have a list of numbers $X = \{x_1, x_2, \dots, x_n\}$, and I expect them to be drawn from a certain distribution. For my case it is the Binomial distribution $P(x) = \binom{n}{x}p^x(1-p)^{n-x}$, but I think I general answer would be the most helpful. What is the standard and most rigorous way I could determine if the samples $X$ are from that distribution?

• You can't determine that observations are from a given distribution. You may sometimes be able to be pretty confident they aren't - and in some circumstances, completely certain of it (e.g. if n=10, none of 11, -5 or 3.4 are even possible), but since something that's not binomial may be arbitrarily close to binomial, no test can tell you that they are binomial. – Glen_b Jul 3 '14 at 15:24
• @GLen_b I disagree somewhat with this position. From a statistical perspective, this is more or less precisely the purpose of equivalence testing. From a philosophy of science perspective, the weakness of a strong falsifiability perspective (which you are, I think, alluding to), is (a) that any hypothesis can be framed in terms of its negative, and (b) there is no objectively preferred null hypothesis. – Alexis Jul 3 '14 at 16:13
• @Alexis I'd politely dispute your use of the word 'precisely'. Equivalence testing answers a different question to the one in the original post. The OP is asking about determining if the samples are binomial. Equivalence testing would be about testing whether they were 'close enough' to binomial by some criterion. The two are logically distinct. I wouldn't disagree that checking for some form of closeness makes more sense. But we don't have some nice TOST procedure for our equivalence test for binomialness, so there's some work to do there. – Glen_b Jul 3 '14 at 16:22
• This is true enough. However, when one uses equivalence tests in combination with difference tests, any difference is no longer meaningful, only relevant difference. So within that inferential framework we sacrifice some precision (i.e. rejecting an equality) for more generality with respect to evidence for and evidence against a target distribution within some tolerance. And yes, some work to do. Possibly related: stats.stackexchange.com/questions/97556/…. – Alexis Jul 3 '14 at 17:04
• I would assume you don't know the parameters of the binomial distribution, but just in case, are they known? – soakley Jul 4 '14 at 1:57

1) You can't determine that observations are from a given distribution.

2) You may sometimes be able to be pretty confident they aren't - and in some circumstances, completely certain of it (e.g. for a binomial, if n=10, none of 11, -5 or 3.4 are even possible, so if you see them, you can reject), but since something that's not binomial may be arbitrarily close to binomial, no test can tell you that they are certainly binomial

Alexis points out the existence of equivalence tests, which could allow you to see whether you have something that's only trivially different ('close enough') from some base case. It might be worth considering these tests, but to my knowledge we don't presently have some nice TOST procedure for our equivalence test for binomialness.

• One can calculate a correlation, and examine an explained fraction ($r^2$). For comparing pdf models, one can compare correlated r-values, and calculate confidence intervals. If the error type is known (e.g., Poisson) one can calculate a noise error and and a misregistration error (e.g., of the square root of the pdf and fit). So, yes, one cannot prove that two things are similar, but one can put bounds on their dissimilarity. – Carl Nov 10 '16 at 21:35
• @Carl You should clarify what you mean ... calculate a correlation between what and what else? The OP has a single sample of size $n$. What's the other variable? – Glen_b Nov 10 '16 at 22:46
• There are two things 1) data as a histogram 2) a model of the histogram as fit to the data. When plotted (for lack of a better term) parametrically, the data histogram and the fit model evaluated at the class mid-points are correlated. How well they correlate is one indication of how good the model is and has two (main) components; histogram noise and model inaccuracy. There is also as slight error from using the midpoints of the class interval data as opposed to the mean or median x-value (best location depending) of that class interval. – Carl Nov 10 '16 at 23:08
• @Carl It's not clear to me what you intend by "data as a histogram". In the question the data are "a list of numbers". The model is binomial, of which the "fit" would consist of a single parameter ($p$) being either estimated or prespecified. You might presume that the list of numbers actually consists of counts (though we don't actually know this for sure) -- is that presumption -- that the $x$'s are counts -- what you intend by "a histogram"? Then one might have a set of expected counts from the binomial (e.g. taking $(\text{expected count at }X=x) = \sum_i x_i\cdot P(x)$). However ...ctd – Glen_b Nov 11 '16 at 0:47
• ctd.... to me a histogram is quite distinct from a list of counts, since a histogram is a manner of displaying certain kinds of counts -- those derived by binning some variable -- and not always the best choice. Note that the question mentions nothing about classes or midpoints. We can take a correlation between expected values and counts, but this can sometimes be a very bad idea (e.g. if the expected values were obtained without reference to the total count in the data, as we might get in some situations I have seen), then the correlation can be nearly 1 but the fit can be terrible. ..ctd – Glen_b Nov 11 '16 at 0:52

Maybe to add some form of more practical answer. One could use the Kolmogorov–Smirnov test which would quantify the distance between the empirical and the reference distribution.

One problem of the KS test is that you would like not to reject the null that that the samples are drawn from the same distribution. Nevertheless, the KS test is rather widely used in a number of different fields.

I think you could further increase the power of the KS test by transforming the original distribution (see here).

Hope this is helping in a more practical sense.

• Aside from the noted problem (that it doesn't do what the question asks, but answers a different question), the tables for the Kolmogorov Smirnov test is based on the assumption of sampling from continuous distributions. If the distribution is discrete (as here) then the test is often highly conservative (with how much depending on the circumstances). – Glen_b Nov 11 '16 at 1:01