Why are Bayesian methods widely considered particularly "convenient"? Several times I have come across (in lectures, papers, etc.) informal/tangential remarks that have the following basic form:

Philosophical issues aside, Bayesian methods are extremely convenient.

...with the implication being that they are more "convenient" (or "handy", "easy", etc.) than competing ("frequentist"?) methods1.  As one can tell from the assertion's preamble, such remarks usually come from people who would not describe themselves as "Bayesians".  (This is what I meant to suggest by the adverb "widely" in this post's title: i.e. even among "non-Bayesians", Bayesian methods are considered particularly convenient.)
What is it about Bayesian methods that make them so convenient?
The best answer to this question I can come up with is that Bayesian methods obviate the need for making some otherwise rather arbitrary decisions about parameters, and whole models.  Instead, the Bayesian formalism serves as a principled way to simultaneously consider entire ensembles of such parameters and models (weighted by their corresponding priors).
Now, my grasp of "Bayesian statistics" is tenuous at best, so I don't put much faith in this explanation.  I would love to see a more informed elaboration.

1 One example I can cite is a recent essay by Gelman and Shalizi's, where the authors disavow the "usual story" (i.e. the conventional interpretation of Bayesian statistics) as their reason for using Bayesian methods, and say instead they use them because with them they can "fit rich enough models" (presumably richer than they could fit using non-Bayesian methods).  Unfortunately, this is the only hard reference I can provide.  The other ones were in the form of informal lecture remarks, or in past reading that I can no longer remember specifically.
 A: In my personal opinion I think that Bayesian methods are extremely convenient in that you need to specify only four main ingredients for solving problems of inference, prediction, and decision making. 
These questions encompass almost all of the discipline of statistics: describing a data set $D$, generalizing outward inferentially from $D$, predicting new data $D^*$, and helping people make decisions in the presence of uncertainty.
Given the set $\mathcal{B}$, of propositions summarizing your background
assumptions and judgments about how the world works as far as $\theta$,
$D$ and future data $D^*$ are concerned:
(a) It's natural (and indeed you must be prepared as a Bayesian) to
specify two conditional probability distributions:


*

*$p(\theta|\mathcal{B})$, to quantify all information about  external to $D$ that You
judge relevant; and

*$p(D|\theta,\mathcal{B})$, to quantify your predictive uncertainty, given $\theta$, about
the data set $D$ before it's arrived.


(b) Given the distributions in (a), the distribution $p(\theta|D,\mathcal{B})$ quantifies
all relevant information about $\theta$, both internal and external to $D$,
and must be computed via Bayes's Theorem:
$$p(\theta|D,\mathcal{B})=c\times p(D|\theta,\mathcal{B})p(\theta|\mathcal{B})\hspace{.75cm}\text{(inference)}$$
where $c > 0$ is a normalizing constant chosen so that the left-hand
side of of the above equation integrates (or sums) over $\Theta$ to 1.
(c) Your predictive distribution $p(D^*|\theta,D,\mathcal{B})$ for future data $D^*$ given the observed data set $D$ must be expressible as follows:
$$p(D^*|D,\mathcal{B})=\int_{\Theta}p(D^*|\theta,D,\mathcal{B})p(\theta|D,\mathcal{B})d\theta$$
often there's no information about $D^*$ contained in $D$ if is known, in which case this expression simplifies to 
$$p(D^*|\mathcal{B})=\int_{\Theta}p(D^*|\theta,D,\mathcal{B})p(\theta|D,\mathcal{B})d\theta\hspace{.75cm}\text{(prediction)}$$
(d) to make a sensible decision about which action $a$ you should take in
the face of your uncertainty about $\theta$, however, you must be prepared to specify
(i) the set $\mathcal{A}$ of feasible actions among which you're choosing, and
(ii) a utility function $U(a,\theta)$, taking values on $\mathbb{R}$ and quantifying your
judgments about the rewards (monetary or otherwise) that would
ensue if you chose action $a$ and the unknown actually took the value $\theta$ (without loss of generality you can take large values of $U(a,\theta$) to be better than small values) then the optimal decision is to choose the action $a$ that maximizes the expectation of $U(a,\theta)$ over $p(\theta|D,\mathcal{B})$, i.e.,
$$a=\arg\max_{a\in\mathcal{A}}\mathbb{E}_{(\theta|D,\mathcal{B})}U(a,\theta)=\arg\max_{a\in\mathcal{A}}\int_{\Theta}U(a,\theta)p(\theta|D,\mathcal{B})d\theta\hspace{.75cm}\text{(decision making)}$$
And thus, there is a very simple and straight forward framework for conducting statistical analysis under the Bayesian framework.  Once again, all you need to specify are the following four ingredients: $p(\theta|\mathcal{B}),p(D|\theta,\mathcal{B})$,the possible actions $a$, and the utility function $U(a,\theta)$. 
Now as convenient as that is, there is no complete guidance under the Bayesian framework that actually tells you how to spefficy those four ingredients and this is actually the subjective and hard part from the Bayesian point of view.  The true convenience of the Bayesian framework comes from the fact that if you are prepared to specify those four ingredients, then there is a clear routine for conducting most all of statistical reasoning. Once again though this is just my opinion.
A: Reasons why I think they're convenient, working in a field that's still somewhat dominated by frequentist statistics, but where a "Bayesian" analysis carry some cachet:


*

*Credible intervals are awesome, in that they can be interpreted the way people really want to interpret confidence intervals from frequentist analysis. This is generally a benefit of Bayesian methods, they're easier to translate into "plain English" statements.

*By "Bayesian" sometimes people mean "MCMC". It's pretty easy to argue that this is incorrect, but a "Bayesian" analysis using MCMC and uninformative priors can occasionally plow through computational problems that likelihood based methods will struggle with.

*A properly done analysis with a very carefully considered set of priors can already set the stage for "How this finding changes the field's understanding of $TOPIC" with no additional input of effort.


That being said, I don't know that I'd actually consider them more convenient than conventional frequentist statistics from an analysis/coding point of view, though well written packages, and things like the BAYES statement in SAS help a great deal.
A: I'm just a rookie in all of this, but it seems to me that the Bayesian "Prior * Likelihood = Posterior" model is simple and flexible, which means "convenient".
For example, your priors are explicitly stated and your procedure does not change because you choose different priors. I could look at your analysis and decide that different priors seem more realistic and could otherwise use the same procedure as you. I may be oversimplifying here, but my feeling about frequentist methods is that they tend to have implicit prior-like assumptions and you choose your method based on what assumptions you accept.
Priors can easily be used to keep your posteriors away from awkward boundary conditions (negative values for counts, for example). The Bayesian procedure is also naturally iterative and mirrors the scientific process: the results of prior experiments/theory become your priors, and the posteriors of your experiment naturally become priors for further experimentation.
Posteriors are convenient because they are not points or even intervals, but rather distributions. This makes it convenient to do various things without complicated follow-on analysis. For example, if you want to determine if two parameters are equal, you can simply look at the differences of their posterior samples, take appropriate quantiles, and decide whether they are equal (difference near zero) or not.
A frequentist test can reject or fail to reject the null. A Bayesian analysis has three possibilities: accept, reject, and not-enough-information. It's convenient when you can actually accept a hypothesis rather than failing to reject it.
As EpiGrad says, Credible Intervals match human intuition and actually answer the question that most of us have. They're simple to calculate and don't require a model to do so: for a 95% CI you take the 0.025 and 0.975 quantiles of your posterior. Simple.
As EpiGrad also says, some people may be confusing MCMC and Bayesian methods, since real-world Bayesina analysis will almost always be MCMC-based. MCMC is a nice, generic black-box procedure, which is convenient -- you could live your whole Bayesian life in BUGS, JAGS, or Stan. Convenient. Though, of course, there can be a lot of tuning and analyzing involved in determining that the MCMC procedure ha actually converged and actually explores the entire density. Which might be the dark underbelly of Bayesian statistics.
