2
$\begingroup$

In a supervised learning context, the posterior distribution of the target given the predictors is often discussed in foundational treatments of the subject. One way this comes up is in decision theory, which makes use of the said posterior distribution to produce the prediction, given some cost function.

In practice, which learning models actually output the posterior distribution? It appears popular decision-tree models and their implementations, such as XGBoost, and neural-network models do not readily output this distribution.

$\endgroup$
4
  • 2
    $\begingroup$ Are you only interested in the posterior distribution of the observable? If so, Tim's answer is by definition correct, since non-Bayesian methods do not have a notion of a "posterior" distribution. However, they do have a notion of a predictive distribution, which aims at the exact same thing, namely a prediction of the conditional distribution of the observable. It's just not arrived at through a Bayesian model or process, so it's not "posterior". $\endgroup$ Nov 20 at 9:20
  • $\begingroup$ @StephanKolassa Thanks for raising this point - the distribution I had in mind is simply conditioned on the inputs/predictors, so no, I'm also interested in non-Bayesian methods that can provide the predictive distribution. $\endgroup$
    – flow2k
    Nov 20 at 23:15
  • 1
    $\begingroup$ @StephanKolassa Bye the way - the terminology on "predictive distribution", not in a posterior/Bayesian sense - which text(s) uses this? $\endgroup$
    – flow2k
    Nov 20 at 23:20
  • 2
    $\begingroup$ Gneiting & Katzfuss (2014), cited in my answer, for instance use this nomenclature. So does Kolassa (2016), apologies for the self-promotion, and I did get it from Gneiting's publications. $\endgroup$ Nov 20 at 23:42
2
$\begingroup$

Only the Bayesian models, that are defined in terms of probability distributions, do. This is by definition, since posterior is obtained using Bayes theorem (see ). Neither of the popular machine learning models does. Moreover, even if there are some Bayesian models implemented in the machine learning software (e.g. BayesianRidge regression in scikit-learn, usually they only produce maximum a posteriori estimates, i.e. find only the mode of the distribution, not the full posterior. For finding posterior distribution, in most cases you would need to implement the models by yourself in probabilistic programming languages like Stan, PyMC, TensorFlow Probability, Pyro, etc. Some of the models would be available out-of-the-box in packages such as R’s brms.

$\endgroup$
2
$\begingroup$

I am most familiar with time series forecasting methods, so my examples will reflect this.

Many, if not most classical forecasting methods will give you a predictive distribution, usually based on a Gaussian assumption, often simply assuming homoskedasticity and estimating the variance in a very naive way. This is where classical ARIMA and exponential smoothing methods get their prediction intervals from. Most forecasting textbooks unfortunately only hint at this.

(G)ARCH is explicitly designed to focus on dynamics in the second moment. It is usually again paired with a normal distribution in estimation, and thus could also be used to derive predictive densities. This is - again implicitly - usually used in Value at Risk forecasting. Nassim Nicholas Taleb essentially claims to have gotten rich off using better distributional assumptions than the normal.

One neural network architecture that explicitly aims at full predictive densities is DeepAR (Salinas et al., 2020). The authors work for Amazon; retailers have a very strong interest in density (or at least quantile) forecasting to set safety amounts. A few years back, Feindt et al. published a little about NeuroBayes, which used neural networks for density forecasting in a Bayesian paradigm; however, it looks like ever since he founded BlueYonder, development has not been published.

Apart from the time series context, of course the simplest example might be OLS and the predictive distributions given by conditional means, (homoskedastic) variance estimates and a t distribution.

A very nice introduction to probabilistic prediction, mainly from the point of view of evaluating such predictions, is Gneiting & Katzfuss (2014).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.