Differences between a frequentist and a Bayesian density prediction What are some essential differences between a frequentist density forecast/prediction and a Bayesian posterior for an outcome of a random variable?
Of course, there will be differences in how they are obtained (via frequentist vs. Bayesian estimation), but I am interested in differences in addition to that. E.g. from a user's perspective, given a frequentist density forecast/prediction vs. a Bayesian posterior, should I treat them differently in any essential way?
 A: In practical terms, there are seven issues that should be thought about with regard to the difference between a Bayesian predictive interval and a Frequentist interval.
The issues are:


*

*Sample size

*Construction

*Boundary conditions

*Coherence

*Information

*Broken intervals

*Interpretation


Each of the above items can either cause a difference in calculation, useability or interpretation.  Of course, the last item is interpretation above.


*

*Generally, for small sample sizes and outside the exponential family of distributions, there is no reason that Bayesian intervals resemble Frequentist intervals.  For some distributions, such as the normal distribution with a diffuse prior, there will be no difference at all in either of the predictive intervals in any practical sense.  For others, such as the Cauchy distribution, you can get pretty wild differences in predictive intervals.

*Construction
2a. Construction of the intervals is on different conceptual grounds.  The Bayesian predictive interval depends on the predictive density function and a rule.  The most common rule in use is to use the highest density region.  This rule corresponds to minimizing the K-L divergence between the model and the future values in nature.  Other rules could also be used as the only requirement is that the prediction adds up to $\alpha{\%}$.  These alternative rules could be understood as minimizing some alternative cost function.
2b. The Frequentist predictive interval depends on a loss function, although the loss function is often implicit.  As with the Bayesian construction, there exists an infinite number of potential prediction intervals because there are an infinite number of potential loss functions.  Frequentist intervals depend upon the sampling distribution of some estimator.  If you change from the sample mean to the sample median you have changed both loss function and sampling distributions.  The predictions will differ.  The parameter estimator vanishes as it does in the Bayesian method.

*Boundary conditions and discreteness do not impact a Bayesian prediction other than it will account for them.  They do impact them in Frequentist methods.  It can happen that a Frequentist interval will include impossible values.  The method also breaks down when using discrete probabilities.
See...

Lawless, J. and Fredette, M. (2005). Frequentist prediction intervals and
  predictive distributions. Biometrika, 92(3):529-542.


*If you need to use the prediction for gambling purposes, such as setting inventory, allocating funds, or playing a lottery then Bayesian intervals are coherent and Frequentist ones are not.  All Frequentist intervals with identical values for their estimators will generate identical intervals though with different samples.  Bayesian prediction intervals, in the general case, will generate different predictive intervals with different samples despite having the same estimator as long as the posterior differs.

*Bayesian predictions are always admissible predictions given a prior and a loss function.  The Bayesian likelihood function is always minimally sufficient.  It is not always the case that a Frequentist method uses as much information and so Frequentist estimators can be noisier given identical information.  For well-behaved models, such as the normal distribution, this is not generally a problem.  Additionally, the Bayesian prediction should include the information in a prior.  If the prior is sufficiently informative, then the Bayesian interval will first-order stochastically dominate the Frequentist interval in terms of loss created by using the prediction in a decision.

*Although this is usually an issue that coincides with small sample sizes or omitted variables, there is no requirement that the Bayesian $\alpha\%$ interval is a single closed interval with a unimodal likelihood.  A Bayesian predictive interval may be $[-5,-1]\cup{[}1,2]$ while the Frequentist interval on the same sample could be $[-2,1]$.  With a bimodal underlying density, there could be broken intervals for either.

*Interpretation
7a. The biggest issue is interpretation.  Assuming valid models for both estimation tools, there are interpretative differences between the intervals.  Frequentist predictive intervals are confidence procedures.  Bayesian intervals might be analogous to credible intervals.  A Frequentist 95% interval will contain future observations at least 95% of the time, with a guarantee of minimal coverage.  There is a 95% chance that a Bayesian 95% interval will contain the future observations.
7b. The Frequentist method guarantees a level of coverage and that it is unbiased,  so it is not a true probability in that it provides no less than an $\alpha\%$ coverage over future predictions.  That is part of what leads to incoherence.  If you need a guarantee of long-run coverage, though not necessarily for the next set of observations, you should use a Frequentist method.  If you need to assign money and minimize the discrepancy between nature and your model, then you should use a Bayesian method.  Do note, however, that Lawless and Fredette's intervals listed above do minimize the average K-L divergence.
For many models that are simple, such as those taught in elementary statistics with an uninformative prior, there is no practical difference except interpretation.  For complex models, they can differ substantively.  You should always think about models in terms of fitness for purpose.  One thing I did leave out, above, which is not a theoretical issue but a practical issue, is computability.  Bayesian methods are notorious for their difficulty in generating a computation of any kind, whereas Frequentist methods often generate a solution in milliseconds.
