How does probabilistic ML "handle uncertainty"? I have heard professors and others say that probabilistic machine learning is useful because it can model or handle uncertainty. I'm not sure what is meant by this. To give an authoritative source, David MacKay writes in his book on inference (p. 531):

Probabilistic modelling also handles uncertainty in a natural manner. It offers a unique prescription, marginalization, for incorporating uncertainty about parameters into predictions...

What is meant by this? How does this handle uncertainty? A comparison to a non-probabilistic model would be appreciated.
 A: Non-probabilistic machine learning models do not handle uncertainty about the parameters. They simply return point estimates for the parameters. You may use additional techniques (e.g. bootstrap) to learn something about the uncertainty. Many of the available solutions (e.g. using dropout also at the prediction time) are thought of as approximations of the Bayesian (probabilistic) solutions to the problem.
Probabilistic models give you estimates of the distributions for the parameters. They tell you what are the probabilities of observing different values of the parameters. This is used to quantify the uncertainty, by calculating things like highest density regions, or quantiles of the distributions.
A: A note on definitions:
From your clarifying comment for examples of what you mean by "probabilistic":

K-means (non-probabilistic) vs. Gaussian mixture model (probabilistic).

It sounds like you're talking about what the literature usually calls parametric vs. non-parametric model. I'd suggest reading [this post][1] from the Machine Learning Mastery blog, and following the references at the bottom of the page under "Posts".
