2
$\begingroup$

I have heard professors and others say that probabilistic machine learning is useful because it can model or handle uncertainty. I'm not sure what is meant by this. To give an authoritative source, David MacKay writes in his book on inference (p. 531):

Probabilistic modelling also handles uncertainty in a natural manner. It offers a unique prescription, marginalization, for incorporating uncertainty about parameters into predictions...

What is meant by this? How does this handle uncertainty? A comparison to a non-probabilistic model would be appreciated.

$\endgroup$
2
  • $\begingroup$ Can you give an example of a probabilistic model and of a non-probabilistic model in your context? I'm not sure if you're asking us to compare models that yield probabilities as classification outcomes vs. models that yield the predicted class directly, or if you're asking a different question like comparing parametric vs non-parametric models. Or something different entirely? $\endgroup$ – Eduard Gelman Jun 5 '18 at 22:03
  • $\begingroup$ K-means (non-probabilistic) vs. Gaussian mixture model (probabilistic). $\endgroup$ – gwg Jun 5 '18 at 23:07
1
$\begingroup$

Non-probabilistic machine learning models do not handle uncertainty about the parameters. They simply return point estimates for the parameters. You may use additional techniques (e.g. bootstrap) to learn something about the uncertainty. Many of the available solutions (e.g. using dropout also at the prediction time) are thought of as approximations of the Bayesian (probabilistic) solutions to the problem.

Probabilistic models give you estimates of the distributions for the parameters. They tell you what are the probabilities of observing different values of the parameters. This is used to quantify the uncertainty, by calculating things like highest density regions, or quantiles of the distributions.

$\endgroup$
2
  • $\begingroup$ I'll accept this, but I want to add the caveat that not all probabilistic methods allow for uncertainty about estimated parameters in the same way. For example, the Bayesian approach (which is what MacKay was talking about) treats the parameter itself as a random variable. However, the frequentist MLE of a statistical model has asymptotic variance associated with the estimator itself. But these aren't really the same thing. My point is that I think this answer is correct for Bayesian inference. $\endgroup$ – gwg Apr 14 at 19:47
  • $\begingroup$ For what it's worth, at the beginning of the book MacKay announces that he'll take a Bayesian slant throughout. $\endgroup$ – Arya McCarthy Apr 15 at 22:21
-1
$\begingroup$

A note on definitions:

From your clarifying comment for examples of what you mean by "probabilistic":

K-means (non-probabilistic) vs. Gaussian mixture model (probabilistic).

It sounds like you're talking about what the literature usually calls parametric vs. non-parametric model. I'd suggest reading [this post][1] from the Machine Learning Mastery blog, and following the references at the bottom of the page under "Posts".

$\endgroup$
4
  • 2
    $\begingroup$ I don't think your distinction between parametric and non-parametric is correct. A non-parametric model is one with a potentially infinite number of parameters. For example, an HMM that grows with your data. Here is a more authoritative source: arxiv.org/abs/1106.2697. And here is an NYU lecture (davidrosenberg.github.io/ml2015/docs/13.mixture-models.pdf) in which the instructor compares $K$-means and GMMs and explicitly states that GMMs are a "Probabilistic Model for Clustering" (Slide 20). $\endgroup$ – gwg Jun 6 '18 at 19:48
  • 2
    $\begingroup$ This is not right. Parametric/non-parametric is not the defining difference here. A gwg points out, non-parametric really means "infinitely parametric." The key distinction between probabilistic and non-probabilistic is whether the parameter values are treated as as points or distributions. Treating them as distributions allows for accounting for "uncertainty" in the prediction. $\endgroup$ – degenerate hessian Jun 6 '18 at 20:18
  • $\begingroup$ I simplified a lot and made some assumptions about what the question was originally about and it looks like I oversimplified my way into being wrong. Accordingly, I've removed the personal commentary. $\endgroup$ – Eduard Gelman Jun 6 '18 at 20:31
  • $\begingroup$ @EduardGelman The edit broke the link. $\endgroup$ – Arya McCarthy Apr 13 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.