Probability that a sample came from a known distribution

Question

I'm looking for a general solution to what I assume must be a common problem because it comes up in every Bayesian calculation, but doesn't seem to be directly answered anywhere. I have an extremely good approximation for a population observation distribution. I took a new sample under an experimental condition. I need to know the probability that the known population distribution produced that new sample, or rather the probability of the sample given the known "null" distribution. I know goodness-of-fit tests are supposed to give p-values for this, but I want to do Bayesian analysis (I need a probability) and the population distribution in this case is not Gaussian.

First, and obviously incorrect, idea: I first thought to sum up the probability of each observation. But, suppose the probability Pr(x=5) = 0.001 and I observe a sample X of size 2000 containing x=5 two thousand times. I know intuitively that such a sample shouldn't have come from distribution Y. If I could just add the probabilities, my probability of Pr(X) = 2000 * Pr(x=5) = 2, which is absurd on so many levels. Maybe I need to multiply or divide that result by some integral in the distribution of Y, or something...

Stated simply, is there any general formula (I don't care how ugly) to calculate the probability of observing a sample when the population distribution is known (of any form)?

Answer 1

The

general formula (I don't care how ugly) to calculate the probability of observing a sample when the population distribution is known

is simply

$$ p(X_1,X_2,\dots,X_n) = \prod_{i=1}^n p(X_i) $$

assuming that $X_1,X_2,\dots,X_n$ are independent. Notice that this does not even assume that they are identically distributed. If they are not independent, the individual probabilities are not enough to infer the joint distribution, as you'd need to know how exactly do they depend on each other.

Answer 2

You have two models, $M_1$ and $M_2$. Each model is characterized by a distribution for an observation $p(y_i|M_j)$. Let $y = (y_1, \ldots, y_n)$. Consistent with an earlier answer, let \begin{equation} p(y|M_j) = \prod_{i=1}^n p(y_i|M_j) . \end{equation} Given the data $y$, this expression provides the likelihood for the model. For Bayesian inference you will also need the prior probabilities for the models, $p(M_j)$, where $p(M_1) + p(M_2) = 1$. Using Bayes rule, the posterior probability of $M_j$ is given by \begin{equation} p(M_j|y) = \frac{p(y|M_j)\,p(M_j)}{p(y)} , \end{equation} where \begin{equation} p(y) = \sum_{j=1}^2 p(y|M_j)\,p(M_j) . \end{equation}