Which predictive models output the posterior distribution? In a supervised learning context, the posterior distribution of the target given the predictors is often discussed in foundational treatments of the subject. One way this comes up is in decision theory, which makes use of the said posterior distribution to produce the prediction, given some cost function.
In practice, which learning models actually output the posterior distribution? It appears popular decision-tree models and their implementations, such as XGBoost, and neural-network models do not readily output this distribution.
 A: Only the Bayesian models, that are defined in terms of probability distributions, do. This is by definition, since posterior is obtained using Bayes theorem (see bayesian). Neither of the popular machine learning models does. Moreover, even if there are some Bayesian models implemented in the machine learning software (e.g. BayesianRidge regression in scikit-learn, usually they only produce maximum a posteriori estimates, i.e. find only the mode of the distribution, not the full posterior. For finding posterior distribution, in most cases you would need to implement the models by yourself in probabilistic programming languages like Stan, PyMC, TensorFlow Probability, Pyro, etc. Some of the models would be available out-of-the-box in packages such as R’s brms.
A: I am most familiar with time series forecasting methods, so my examples will reflect this.
Many, if not most classical forecasting methods will give you a predictive distribution, usually based on a Gaussian assumption, often simply assuming homoskedasticity and estimating the variance in a very naive way. This is where classical ARIMA and exponential smoothing methods get their prediction intervals from. Most forecasting textbooks unfortunately only hint at this.
(G)ARCH is explicitly designed to focus on dynamics in the second moment. It is usually again paired with a normal distribution in estimation, and thus could also be used to derive predictive densities. This is - again implicitly - usually used in Value at Risk forecasting. Nassim Nicholas Taleb essentially claims to have gotten rich off using better distributional assumptions than the normal.
One neural network architecture that explicitly aims at full predictive densities is DeepAR (Salinas et al., 2020). The authors work for Amazon; retailers have a very strong interest in density (or at least quantile) forecasting to set safety amounts. A few years back, Feindt et al. published a little about NeuroBayes, which used neural networks for density forecasting in a Bayesian paradigm; however, it looks like ever since he founded BlueYonder, development has not been published.
Apart from the time series context, of course the simplest example might be OLS and the predictive distributions given by conditional means, (homoskedastic) variance estimates and a t distribution.
A very nice introduction to probabilistic prediction, mainly from the point of view of evaluating such predictions, is Gneiting & Katzfuss  (2014).
