Sampling uncertainty of posterior probability distribution I'm working on a problem with 3 possible outcomes and a bunch of features. I have a regression model that outputs probabilities for each category and I'd like to extend these probabilities to probability distributions. So instead of getting an output of [0.2,0.3,0.5], I'd get a probability density function or at least a quantification of the uncertainty of the prediction. I've looked into some models that give these distributions but haven't found any way to use it on a multiple output problem. Is there any way to do this?
 A: Interesting problem - which is most often overlooked in data science and machine learning. The output probabilities $\bf{y}$ are indeed estimates of the underlying (true) posterior probabilities (your $[0.2,0.3,0.5]$). Sampling a different training set (from your presupposed 'oracle'), will yield a slightly different set of output probabilities, when the identical input feature vector $\bf{x}$ is presented to the classifier.
The distributions of $\hat{P}(\bf{y} \mid \bf{x},\bf{\theta})$ - they have been studied for linear and quadratic discriminant analysis ($\theta$ is the parameter vector of the discriminant classifier).
And yes, also the sufficient parameters of these distributions of $\hat{P}(\bf{y} \mid \bf{x},\bf{\theta})$ have been derived. Specifically the variance of each posterior probability has been derived. A mathematically sound description (with the relevant references to papers in the statistical literature), can be found in Chapter 11 in the book: Discriminant analysis and statistical pattern recognition by G.J. McLachlan, Wiley (2004).
