**Edits:** Reorganized and expanded my answer to make better sense.

I am giving an answer as a *naive* empirical Bayesian here.

First, the posterior distribution allows you to do computations that you simply cannot do with a straightforward MLE.
The simplest case is that *today's posterior is tomorrow's prior*. Bayesian inference naturally allows for sequential updates, or more in general online or delayed combination of multiple sources of information (incorporating a prior is just one textbook instance of such combination).
Bayesian Decision Theory with a nontrivial loss function is another example. I would not know what to do otherwise.

Second, with this answer I will try and argue that the mantra that *quantification of uncertainty is generally better than no uncertainty* is effectively an empirical question, since theorems (as you mentioned, and as far as I know) provide no guarantees.

### Optimization as a toy model of scientific endeavor

A domain that I feel fully captures the complexity of the problem is a very practical, no-nonsense one, the optimization of a *black-box* function $f: \mathcal{X} \subset \mathbb{R}^D \rightarrow \mathbb{R}$. We assume that we can sequentially query a point $x \in \mathcal{X}$ and get a possibly noisy observation $y = f(x) + \varepsilon$, with $\varepsilon \sim \mathcal{N}(0,\sigma^2)$. Our goal is to get as close as possible to $x^* = \arg\min_x f(x)$ with the minimum number of function evaluations.

A particularly effective way to proceed, as you may expect, is to build a *predictive model* of what would happen if I query any $x^\prime \in \mathcal{X}$, and use this information to decide what to do next (either locally or globally). See [Rios and Sahinidis (2013)](http://thales.cheme.cmu.edu/dfo/comparison/dfo.pdf) for a review of derivative-free global optimization methods.
When the model is complex enough, this is called a *meta-model* or *surrogate-function* or *response surface* approach. Crucially, the model could be a point estimate of $f$ (e.g., the fit of a radial basis network function to our observations), or we could be Bayesian and somehow get a full posterior distribution over $f$ (e.g., via a Gaussian process).

[Bayesian optimization](https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf)  uses the posterior over $f$ (in particular, the joint conditional posterior mean and variance at any point) to guide the search of the (global) optimum via some principled heuristic. The classical choice is to maximize the *expected improvement* over the current best point, but there are even fancier methods, like minimizing the expected entropy over the location of the minimum (see also [here](https://stats.stackexchange.com/questions/193306/optimization-when-cost-function-slow-to-evaluate/193391#193391)).

The empirical result here is that having access to a posterior, even if partially misspecified, generally produces better results than other methods.
(There are caveats and situations in which Bayesian optimization is no better than random search, such as in high dimensions.)

Since you asked -- this has a much higher computational cost than other non-Bayesian methods, and you were wondering why we should be Bayesian. 
The assumption here is that the cost involved in evaluating the true $f$ (e.g., in a real scenario, a complex engineering or machine learning experiment) is much larger than the computational cost for the Bayesian analysis, so being Bayesian *pays off*.

### What can we learn from this example?

First, why does Bayesian optimization work at all? I guess that the model is wrong, but not *that* wrong, and as usual *wrongness* depends on what your model is for. For example, the exact shape of $f$ is not relevant for optimization, since we could be optimizing any monotonic transformation thereof. I guess nature is full of such invariances.
So, the search we are doing might not be optimal (i.e., we are throwing away good information), but still better than with no uncertainty information.

Second, our example highlights that it is possible that the usefulness of being Bayesian or not *depends on the context*, e.g. the relative cost and amount of available (computational) resources. (Of course if you are a hardcore Bayesian you believe that [every computation is Bayesian inference](https://arxiv.org/abs/1506.01326) under some prior and/or approximation.)

Finally, the big question is -- why are the models we use *not-so-bad* after all, in the sense that the posteriors are still useful and not statistical garbage? If we take the No Free Lunch theorem, apparently we shouldn't be able to say much, but luckily we do not live in a world of *completely random* (or *adversarially chosen*) functions. 

More in general, since you put the "philosophical" tag... I guess we are entering the realm of the problem of induction, or the unreasonable effectiveness of mathematics in the *statistical* sciences (specifically, of our mathematical intuition & ability to specify models that work in practice) -- in the sense that from a purely a priori standpoint there is no reason why our guesses should be good or have any guarantee (and for sure you can build mathematical counterexamples in which things go awry), but they turn out to work well in practice.