Probability that a sample came from a known distribution

I'm looking for a general solution to what I assume must be a common problem because it comes up in every Bayesian calculation, but doesn't seem to be directly answered anywhere. I have an extremely good approximation for a population observation distribution. I took a new sample under an experimental condition. I need to know the probability that the known population distribution produced that new sample, or rather the probability of the sample given the known "null" distribution. I know goodness-of-fit tests are supposed to give p-values for this, but I want to do Bayesian analysis (I need a probability) and the population distribution in this case is not Gaussian.

First, and obviously incorrect, idea: I first thought to sum up the probability of each observation. But, suppose the probability Pr(x=5) = 0.001 and I observe a sample X of size 2000 containing x=5 two thousand times. I know intuitively that such a sample shouldn't have come from distribution Y. If I could just add the probabilities, my probability of Pr(X) = 2000 * Pr(x=5) = 2, which is absurd on so many levels. Maybe I need to multiply or divide that result by some integral in the distribution of Y, or something...

Stated simply, is there any general formula (I don't care how ugly) to calculate the probability of observing a sample when the population distribution is known (of any form)?

The

general formula (I don't care how ugly) to calculate the probability of observing a sample when the population distribution is known

is simply

$$p(X_1,X_2,\dots,X_n) = \prod_{i=1}^n p(X_i)$$

assuming that $$X_1,X_2,\dots,X_n$$ are independent. Notice that this does not even assume that they are identically distributed. If they are not independent, the individual probabilities are not enough to infer the joint distribution, as you'd need to know how exactly do they depend on each other.

You have two models, $$M_1$$ and $$M_2$$. Each model is characterized by a distribution for an observation $$p(y_i|M_j)$$. Let $$y = (y_1, \ldots, y_n)$$. Consistent with an earlier answer, let $$$$p(y|M_j) = \prod_{i=1}^n p(y_i|M_j) .$$$$ Given the data $$y$$, this expression provides the likelihood for the model. For Bayesian inference you will also need the prior probabilities for the models, $$p(M_j)$$, where $$p(M_1) + p(M_2) = 1$$. Using Bayes rule, the posterior probability of $$M_j$$ is given by $$$$p(M_j|y) = \frac{p(y|M_j)\,p(M_j)}{p(y)} ,$$$$ where $$$$p(y) = \sum_{j=1}^2 p(y|M_j)\,p(M_j) .$$$$

• For deciding p(Mj): If you have n models and you're unsure about the probability that a given model is the right one, you can assign a uniform prior belief to every model, i.e., p(Mj) = 1/n (in this case that would be 0.5 for both models). This may sound subjective - and it is. Bayesian statistics starts with making assumptions (priors) about the probability of each model, and then using those assumptions to calculate a p(Mj | y). If you get new data from the same source and want to check the model probabilities again, you can use the old p(Mj | y) values you calculated as the new prior. Commented Mar 10, 2022 at 19:40