When is MCMC required? I am trying to understand when MCMC algorithms are utilized. I can do density estimation with MLE for non-linear cases just like linear cases, and EM for hidden variables and missing values. What are the scenarios when these tools are not sufficient and we use MCMC?
Thanks
 A: Got too long for a comment, so I guess it has to be an answer.
Is MCMC ever actually required? I don't think so. There's nearly always some way of proceeding without it.
Sometimes, one or another MCMC algorithm is quite convenient, where alternatives are not so convenient. 
Here's one example:
To my mind a big part of what makes MCMC especially convienient - particularly with things like Gibbs and variations (Metropolis-within-Gibbs, block Gibbs etc) is the ability to construct components of models piece by piece as if everything else were okay. So you might have a tool for variable selection or selection of model order, and you might have a tool for dealing with outliers, and you might have a tool for dealing with estimation and you might have a tool for dealing with changing variance, but such procedures usually rely on everything else being either absent, or known rather than estimated.
What makes an outlier an outlier depends on your model, ... but what are the properties of your variable selection if you haven't controlled for outliers? If you do first one and then the other and then some of the first again... what are the properties of that?
MCMC often lets you work with those components conditionally on knowing the other things that would be inconvenient if you didn't know them (while still allowing you to integrate out the things it's convenient to integrate out). If you can establish that the particular details of the sampler you use converges to the desired stationary distribution (trivial if you do something standard, usually not difficult otherwise) and you actually iterate to effective convergence to it (rather harder to assess), then you are sampling from the stationary distribution (in a Bayesian context, typically the desired joint distribution of all your unknowns, from which you can extract the marginals).
That's by no means even remotely exhaustive - for example, I haven't touched on the quite different kind of convenience that Random Walk Metropolis-Hastings can give.
The best way to really begin to understand the value in MCMC is to use it on a few real problems that are otherwise relatively hard.
A: MCMC changed Bayesian inference forever, allowing us to go well beyond inference based on conjugate priors. We may see MCMC as a set of tools that allows us to explore any posterior distribution, even when we don't know the analytic expression of its normalization constant (this is analogous to those situations in Physics in which we don't have an analytic expression for the partition function). MCMC gives you a sample of (generally dependent) parameter values from the posterior distribution. From this sample, with the help of the ergodic theorem, any posterior summary can be computed, such as the posterior expectation, a credible interval (more generally, a credible set), or the posterior probability of events involving any (measurable) transformation of the parameters. The existence of a plethora of MCMC methods is due to the fact that, although algorithms like Metropolis-Hastings are pretty much universal, particular algorithms may perform better in specific scenarios. To develop a taste for the need and importance of MCMC, I would do all the possible computations with conjugate priors (books by Robert, Lee, De Groot and Schervish, and others are good starting points), and then proceed slowly to more general problems (Robert and Casella is the best MCMC reference that I have seen so far).
