I am probably being a naive empirical Bayesian here. I am going with the mantra that quantification of uncertainty is generally better than no uncertainty, which may sound like a theoretical dogma but I found out to hold empirically (somewhat to my surprise).
The posterior distribution allows you to do computations that you simply cannot do with a straightforward MLE. The simplest case is that today's posterior is tomorrow's prior. Bayesian inference naturally allows for sequential updates, or more in general online or delayed combination of multiple sources of information (incorporating a prior is just one textbook instance of such combination). Bayesian Decision Theory with a nontrivial loss function is another example. I would not know what to do otherwise.
This is all good and known but the specific example I have in mind to make my case (and that convinced me that being Bayesian is a good thing, not just a faction that I happened to like) is Bayesian optimization, see e.g.,
Shahriari, Bobak, et al. "Taking the human out of the loop: A review of Bayesian optimization." Proceedings of the IEEE 104.1 (2016): 148-175.
or more in general probabilistic numerics,
Hennig, Philipp, Michael A. Osborne, and Mark Girolami. "Probabilistic numerics and uncertainty in computations." Proc. R. Soc. A. Vol. 471. No. 2179. The Royal Society, 2015.
In the case of Bayesian optimization, the key idea is to have a posterior over a function (e.g., via a Gaussian process) as opposed to a point estimate (e.g., as you would get by fitting a radial basis function) and to use that posterior -- in particular, the conditional posterior mean and variance at each point -- to guide the search of the (global) optimum via some principled heuristic (the classical choice is to maximize the expected improvement over the current best point, but there are even fancier methods, like minimizing the expected entropy over the location of the minimum).
Crucially, having access to a posterior, even if (largely) misspecified, seems to produce better empirical results in a very practical, no-nonsense endeavor such as optimizing a function. (There are caveats and situations in which Bayesian optimization is no better than random search, such as in high dimensions.)
So, what's going on here? I guess that the underlying assumption here is that the model is wrong, but not that wrong, such that the answers that we get might be suboptimal (i.e., we might be throwing away good information), but still better than with no uncertainty information.
Why does it work at all? I don't have an answer. I guess we are entering the realm of the problem of induction, or the unreasonable effectiveness of mathematics in the statistical sciences (specifically, of our mathematical intuition & ability to specify models that work in practice) -- in the sense that from a purely a priori standpoint there is no reason why our guesses should be good or have any guarantee (and for sure you can build mathematical counterexamples in which things go awry), but they turn out to work well in practice.