I was referring to this video lecture http://videolectures.net/mlss09uk_murray_mcmc/ where the speaker had something like this
$$ p(x|D) = \int P(x|\theta,D)P(\theta,D)d\theta $$
equivalent to
$$ \frac{1}{S}\sum_{s=1}^S P(x|\theta^{(s)},D), \theta^{(s)} \sim P(\theta|D) $$
I didn't get how the average was derived and why is $\theta^{(s)} \sim P(\theta|D) $ Can anyone explain?
May I use a slightly different (but still sloppy, confusing random objects with their realizations, etc.) notation? I didn't watch the lecture, but it seems that you know the value of $n$ observations $(x_1,\dots,x_n)=x^{(n)}$, and you are interested in the predictive density $f(x_{n+1}\mid x^{(n)})$. Marginalizing, using the product rule, and using the fact that $x_1,\dots,x_n,x_{n+1}$ are conditionally iid given $\theta$, we have $$ f(x_{n+1}\mid x^{(n)}) = \int f(x_{n+1},\theta\mid x^{(n)})\,d\theta = \int f(x_{n+1}\mid\theta,x^{(n)})\,\pi(\theta\mid x^{(n)})\,d\theta $$ $$ = \int f(x_{n+1}\mid\theta)\,\pi(\theta\mid x^{(n)})\,d\theta = (*) $$ Please, be sure that you understand the three equalities above. The last integral is the integral of "stuff" times a density (the posterior), so it is the expectation of "stuff" $$ (*) = \mathbb{E}\left[f(x_{n+1}\mid\Phi)\right] \, , $$ where the distribution of $\Phi$ has density $\pi(\,\cdot\mid x^{(n)})$.
Suppose that you have a sequence of iid random variables $\Phi_1,\Phi_2,\dots$, such that $\Phi_i\sim\Phi$. By the strong law of large numbers $$ \frac{1}{N} \sum_{i=1}^N f(x_{n+1}\mid\Phi_i) \to \mathbb{E}\left[f(x_{n+1}\mid\Phi)\right] \, , $$ almost surely, as $N\to\infty$.
Actually, if you are doing MCMC, your sequence of $\Phi_i$'s (a Markov chain) will be a dependent sequence, but given some regularity conditions that the chain satisfies, the Ergodic Theorem guarantees that you have the almost sure convergence described above.
