Advantages of Bayesian Methods for Parameter Estimation Let us suppose that I have data that is normally distributed. I wish to find the $\mu$ and $\sigma$ parameters. The common sense thing to do is to simple calculate the sample mean and deviation. No fancy math, be done with it, and understand that a margin-of-error is expected. If we wish to quantify the margin of error we can use a confidence interval.
However, the fancy Bayesian approach is instead provide probability distributions for the two parameters. What benefit does this bring? I imagine there are problems were Bayesian approaches are better, but in this simplistic example that was mentioned what is the benefit?
 A: One obvious benefit is that given a probability distribution and an evaluation* loss function, you can derive the optimal point estimate, i.e. one that minimizes the expected loss. This is not strictly specific to Bayesian methods but to any method that delivers a probability distribution instead of a point or an interval estimate. However, Bayesian methods are perhaps the most popular among such methods (another option would be fiducial methods).
Another benefit is the ability to seamlessly incorporate useful prior information into the estimate. If you have (strong) prior information, your Bayesian estimate will frequently be more accurate than, say, a frequentist estimate.
A third benefit concerns interpretation. Bayesian methods tell us something about the unknown parameter conditional on the observed data (something that is often of direct interest) rather than about the observed data conditional on the unknown parameter (often not directly relevant). The interpretation of a Bayesian result is thus often more straightforward.
*See here for details of what evaluation loss means and how it differs from estimation loss.
