I have a probability distribution. And I have a set of values. I need to figure out how to calculate the probability that these values were generated by the same model as the distribution.

I found this answer: Probability that a sample came from a known distribution

It does not make sense to me, based on a simple thought experiment that I do:

Lets say there are 2000 identical values. According to the distribution, this value has a 0.99 chance of being drawn. Rationally, it would make sense that this set of 2000 values came from this distribution. However, using the above answer the probability that the values came from the distribution is given by 0.99^2000 which is 1.86 * 10^-9

So, how do I calculate the probability that a set of values came from a distribution?

Personal attempts: I tried adjusting with a number of permutations, however the answer stopped making sense in the reverse thought experiment. Having a set of 2000 identical values, each of which have a probability of 0.1 of being drawn, the probability of the set coming from the distribution is then 0.1^2000 * 2000! which is 3.31 * 10 ^ 3735. So this also doesn't make sense.

Personal answer:

I think what I'm actually looking for is a z-test or a t-test, but generalized to any distribution. Thus the approach would be to find the mean of the sample and compare it to the mean of the distribution.

The difficulty is that depending on the size of the sample, the probability of the sample mean being a D distance away from the distribution mean changes.

How to calculate this probability becomes the question. I do not see any other way but to use Markov Chain approach, and use computation repetitively create random samples of size S using the known distribution and create a distribution of distance D for sample size S. Then this distribution can be used to calculate the probability of the sample mean being different to the mean original distribution/model.

  • $\begingroup$ Are we restricting consideration to discrete distributions here? $\endgroup$ – Glen_b Oct 13 '19 at 0:38
  • $\begingroup$ We could. I am doing some computations, so it is easiest (on the brain at least) to make them descrete. $\endgroup$ – Anton Oct 14 '19 at 11:41
  • 2
    $\begingroup$ Let's examine your hypothetical a little more closely. Suppose the underlying distribution were Normal. Generate independently 2000 values from it. The probability of that sample is zero. The point is that you are confusing two probabilities: the chance of obtaining a sample from a distribution versus the chance a distribution was the source of a sample. The latter cannot be computed until you (a) stipulate all the possible source distributions and (b) provide a probability distribution for them (a "distribution of distributions," aka a prior distribution). $\endgroup$ – whuber Oct 14 '19 at 16:20

"Rationally, it would make sense that this set of 2000 values came from this distribution". Why? If the probability of getting this value is 0.99 then in 100 outcomes you'd expect one different value. But you are getting 2000 identical values! Isn't it odd?

Try with the probability of getting this value being 0.9999 instead of 0.99. Now 0.9999^2000=0.82.

Think like this: having observed 2000 outcomes all equal to $x$, you are trying to figure out if these observations came from, say, $X$ where $Pr(X=x)=0.99$. In absolute terms you get a very low probability of this happening. But compare to what? This distribution is still a better fit than, say, $Y$ where $Pr(Y=x)=0.9$ even as it is worse than, say, $Z$ where $Pr(Z=x)=0.9999$. So the result of your thought experiment is right. It's just that the probability has come lower than you expected.

  • $\begingroup$ what do you think of my answer? $\endgroup$ – Anton Oct 14 '19 at 15:36
  • $\begingroup$ Initially I was also thinking of suggesting for t-test but as pointed out by whuber that is a different probability. The probability of obtaining a particular sample even in discrete case would always be very low. Coming to your answer, D is captured in the variance of sample mean. Since sample mean follows Normal due to CLT, the probability of D can be easily calculated. The problem you will again face is that for a given D the probably will again be very small/zero. That's why we construct confidence intervals. $\endgroup$ – Dayne Oct 15 '19 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.