2
$\begingroup$

I'm working on a problem with 3 possible outcomes and a bunch of features. I have a regression model that outputs probabilities for each category and I'd like to extend these probabilities to probability distributions. So instead of getting an output of [0.2,0.3,0.5], I'd get a probability density function or at least a quantification of the uncertainty of the prediction. I've looked into some models that give these distributions but haven't found any way to use it on a multiple output problem. Is there any way to do this?

$\endgroup$
  • $\begingroup$ Only your title is too general. You can change this to something like: sampling uncertainty of posterior probability distribution - or something similar, but more specific than your present title. $\endgroup$ – Match Maker EE Jul 24 at 23:41
2
$\begingroup$

Interesting problem - which is most often overlooked in data science and machine learning. The output probabilities $\bf{y}$ are indeed estimates of the underlying (true) posterior probabilities (your $[0.2,0.3,0.5]$). Sampling a different training set (from your presupposed 'oracle'), will yield a slightly different set of output probabilities, when the identical input feature vector $\bf{x}$ is presented to the classifier.

The distributions of $\hat{P}(\bf{y} \mid \bf{x},\bf{\theta})$ - they have been studied for linear and quadratic discriminant analysis ($\theta$ is the parameter vector of the discriminant classifier).

And yes, also the sufficient parameters of these distributions of $\hat{P}(\bf{y} \mid \bf{x},\bf{\theta})$ have been derived. Specifically the variance of each posterior probability has been derived. A mathematically sound description (with the relevant references to papers in the statistical literature), can be found in Chapter 11 in the book: Discriminant analysis and statistical pattern recognition by G.J. McLachlan, Wiley (2004).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.