I have a probability distribution. And I have a set of values. I need to figure out how to calculate the probability that these values were generated by the same model as the distribution.
I found this answer: Probability that a sample came from a known distribution
It does not make sense to me, based on a simple thought experiment that I do:
Lets say there are 2000 identical values. According to the distribution, this value has a 0.99 chance of being drawn. Rationally, it would make sense that this set of 2000 values came from this distribution. However, using the above answer the probability that the values came from the distribution is given by
1.86 * 10^-9
So, how do I calculate the probability that a set of values came from a distribution?
I tried adjusting with a number of permutations, however the answer stopped making sense in the reverse thought experiment. Having a set of 2000 identical values, each of which have a probability of
0.1 of being drawn, the probability of the set coming from the distribution is then
0.1^2000 * 2000! which is
3.31 * 10 ^ 3735. So this also doesn't make sense.
I think what I'm actually looking for is a z-test or a t-test, but generalized to any distribution. Thus the approach would be to find the mean of the sample and compare it to the mean of the distribution.
The difficulty is that depending on the size of the sample, the probability of the sample mean being a
D distance away from the distribution mean changes.
How to calculate this probability becomes the question. I do not see any other way but to use Markov Chain approach, and use computation repetitively create random samples of size
S using the known distribution and create a distribution of distance
D for sample size
S. Then this distribution can be used to calculate the probability of the sample mean being different to the mean original distribution/model.