Checking the goodness of a probability model I have a data set that I claim is adequately modeled by a probability distribution, or more generally a family of probability distributions. How do you decide that the model is good/useful/correct under the stipulation that you can only refer to this data set? 
For concreteness, the data set could be [-3,2,1,0,10020] and the proposed model for an individual data point is a Normal(theta,1) where theta is real; additionally, the data points are independent.
 A: You can chart a reliability diagram (DeGroot, M., & Fienberg, S. (1982). The comparison and evaluation of forecasters. Statistician, 32, 12–22).
Have a look at http://www.datascienceassn.org/sites/default/files/Predicting%20good%20probabilities%20with%20supervised%20learning.pdf
"On real problems where the true conditional probabilities
are not known, model calibration can be visualized with reliability
diagrams. First, the prediction space is discretized into ten bins. Cases with
predicted value between 0 and 0.1 fall in the first bin, between
0.1 and 0.2 in the second bin, etc. For each bin, the
mean predicted value is plotted against the true fraction of
positive cases. If the model is well calibrated the points
will fall near the diagonal line."
Here is an example I could find:

(source: bom.gov.au)
Hope this helps.
A: Ordinarily, when we fit data to a proposed model, we test whether the model is "adequate" by looking at the distribution of the residual values, and comparing aspects of this distribution to the theoretical distribution that holds under the proposed model.  This process is called "diagnostic testing".  There are many diagnostic tests that you can run on the residuals, depending on what aspect of their distribution you want to test.  Diagnostic testing is done with a mixture of graphical analysis and formal hypothesis testing.

Example: In your question, you give the example of the observed data set $\mathbf{x} = (-3,2,1,0,10020)$ fit to the proposed model $X_i \sim \text{IID N}(\theta, 1)$.  Now, just looking at this, I can see it is an inadequate model, since the sample data has a much higher variance than is allowed by the model.  This is going to lead all your data points to be classified as "outliers" in the model, meaning that the model underspecifies either the variance or the fatness of the tails.  (In this case, it underspecifies the variance.)
To test this formally, using diagnostic testing, let's suppose that we estimate the unknown mean parameter in the model by the MLE, which is the sample mean $\hat{\theta} = \bar{x} = 2004$.  With this estimate of the mean parameter, the residuals are:
$$\mathbf{r} = (-2007, -2002, -2003, -2004, +8016).$$
Now, under the proposed model, we have the distribution $||\mathbf{R}|| \sim \chi_{n-1}$ for the norm of the residual vector.  The observed norm of the residual vector is $||\mathbf{r}|| = 8962$, which corresponds to a tail area of essentially zero.  (The observed norm is over 40 million standard deviations above the mean under the proposed model.)  This diagnostic test shows that the proposed model is implausible, insofar as it fixed a variance that is too low.  We would reject this model and use a model that allows a free variance parameter.
A: You can calculate the likelihood (probability of the data, given the model with its parameters). Or log likelihood, which is more practical to compute.
