When is MCMC useful? I am having trouble in understanding in which situation the MCMC approach is actually useful. I am going through a toy example from the Kruschke book "Doing Bayesian Data Analysis: A Tutorial with R and BUGS".
What I understood so far is that we need a target distribution which is proportional to $p(D|\theta)p(\theta)$ in order to have a sample of $P(\theta|D)$. However, it seems to me that once we have $p(D|\theta)p(\theta)$ we only need to normalize the distribution to get the posterior, and the normalization factor could be easily found numerically.
So what are the cases when this is not possible? 
 A: When you are given a prior $p(\theta)$ and a likelihood $f(x|\theta)$ that are either not computable in closed form or such that the posterior distribution $$p(\theta|x)\propto p(\theta)f(x|\theta)$$is not of a standard type, simulating directly from this target towards a Monte Carlo approximation of the posterior distribution is not feasible. A typical example is made of hierarchical models with non-conjugate priors, such as those found in the BUGS book. 
Indirect simulation methods such as accept-reject, ratio-of-uniform, or importance-sampling techniques customarily run into numerical and precision difficulties when the dimension of the parameter $\theta$ increases beyond a few units.
On the opposite, Markov chain Monte Carlo methods are more ameanable to large dimensions in that they can explore the posterior distribution on a local basis, i.e. in a neighbourhood of the current value, and on a smaller number of components, i.e., on subspaces. For instance, the Gibbs sampler validates the notion that simulating from a one-dimensional target at a time, namely the full conditional distributions associated with $p(\theta|x)$, is sufficient to achieve simulation from the true posterior in the long run.
Markov chain Monte Carlo methods also some degree of universality in that algorithms like the Metropolis-Hastings algorithm is formally available for any posterior distribution $p(\theta|x)$ that can be computed up to a constant.
In cases when $p(\theta)f(x|\theta)$ cannot be easily computed, alternatives exist, either by completing this distribution into a manageable distribution over a larger space, as in$$p(\theta)f(x|\theta)\propto \int g(z|\theta,x)
p(\theta)f(x|\theta)\text{d}z$$ or through non-Markovian methods like ABC.
MCMC methods have given a much broader reach for Bayesian methods, as illustrated by the upsurge that followed the popularisation of the method by Alan Gelfand and Adrian Smith in 1990.
A: Monte Carlo integration is one form of numerical integration which can be much more efficient than, e.g., numerical integration by approximating the integrand with polynomials. This is especially true in high dimensions, where simple numerical integration techniques require large numbers of function evaluations. To compute the normalization constant $p(D)$, we could use importance sampling,
$$p(D) = \int \frac{q(\theta)}{q(\theta)} p(\theta)p(D \mid \theta) \, d\theta
\approx \frac{1}{N} \sum_n w_n p(\theta_n)p(D \mid \theta_n),$$
where $w_n = 1/q(\theta_n)$ and the $\theta_n$ are sampled from $q$. Note that we only need to evaluate the joint distribution at the sampled points. For the right $q$, this estimator can be very efficient in the sense of requiring very few samples. In practice, choosing an appropriate $q$ can be difficult, but this is where MCMC can help! Annealed importance sampling (Neal, 1998) combines MCMC with importance sampling.
Another reason why MCMC is useful is this: We usually aren't even that interested in the posterior density of $\theta$, but rather in summary statistics and expectations, e.g.,
$$\int p(\theta \mid D) f(\theta) \, d\theta.$$
Knowing $p(D)$ does not generally mean we can solve this integral, but samples are a very convenient way to estimate it.
Finally, being able to evaluate $p(D \mid \theta)p(\theta)$ is a requirement for some MCMC methods, but not all of them (e.g., Murray et al., 2006).
