In frequentist analysis, we define a 95% prediction interval as an interval that will contain the next observation 95% of the time under repeated sampling of the entire experiment and prediction. If we are dealing with a scalar observable, we may even be able to turn this into a type of "predictive distribution" (in the Fiducial sense).
This all makes sense to me, since the interval and/or distribution are not purporting to be the actual distribution of the next point, only devices for making predictions with the correct operating characteristics.
In contrast, I've just finished reading Gelman et al's Bayesian Data Analysis (3rd ed) and I have an issue with how to interpret the posterior predictive intervals that are calculated for posterior predictive checks. In Dr. Gelman's view, our prior + likelihood both constitute our stochastic model of the data generating process. In this model, "nature" first draws a value for the true parameter, then generates observations using this parameter. Note: For the rest of my quesion, please remember that the Bayesian model assumes nature did this only once, to fix our parameter, and then started generating values.
Now, let's say we want to compare our fitted model by comparing the distribution of our model to the actual values from our experiment (say, of size 100). To do this, Gelman suggests we simply generate $M$ draws from the posterior distribution of the parameters and then, for each draw, draw a sample of 100 values from the associated data model (likelihood). This is essentially assuming that each sample of 100 observations gets its own (different) parameter from the posterior distribution; therefore, the resulting distribution will generally be wider than if we simulated future values using the known (true) parameter value that actually generated the data.
And here's where I have in issue: Let's say I plan to take 10,000 more observations from the same process (i.e., predict what I would have seen if I had extended the experiment beyond 100 observations). In this case, it's provably true that these 10,000 observations will not follow the posterior predictive distribution, since the posterior predictive distribution includes variability/randomness in the underlying parameter, but the actual data are being generated from a fixed parameter (see my note from above). Hence, posterior predictive distributions do not give an accurate picture of the distribution of future observations.
So, if this is the case, how can comparing the observed data (which does not actually have parameter variability) with the fitted posterior predictive distribution help us except to see gross errors? Even if our Bayesian model were 100% correct, the posterior predictive distribution would still be wider than the true data distribution.
Can someone explain what a posterior predictive distribution is actually modeling?
Perhaps I'm interpreting the posterior predictive distribution too literally (but I could be forgiven given it's use by Gelman). Maybe the "posterior predictive distribution" is simply a measure over the range of the observations that allows you to assign a probability to an interval. It implies that since we don't know the true value of the parameter, we are averaging over all possible models in proportion to their posterior probability.
In this way, posterior predictive distributions are closer to a mixture distribution, than to the true distribution of the data: the distribution "averages over" possible futures, but will not necessarily reflect any particular future (a large sample of new observations will not look like the posterior predictive distribution).
This has a repeated sampling interpretation: if we repeatedly drew new experiments from our Bayesian model, conditional on getting the exact same observations we actually did, then took an additional 1000 observations each time, a 90% posterior prediction interval will contain approximately 90% of these observations on average. This is just like frequentist prediction intervals.