# Confusion related to prediction

I was referring to this video lecture http://videolectures.net/mlss09uk_murray_mcmc/ where the speaker had something like this

$$p(x|D) = \int P(x|\theta,D)P(\theta,D)d\theta$$

equivalent to

$$\frac{1}{S}\sum_{s=1}^S P(x|\theta^{(s)},D), \theta^{(s)} \sim P(\theta|D)$$

I didn't get how the average was derived and why is $\theta^{(s)} \sim P(\theta|D)$ Can anyone explain?

-
Do you recognize the formula for $p(x|D)$ as an expectation and the set of $\theta^{(s)}$ as an iid sample? If so, you might see that the two expressions are not precisely equivalent, but that the latter estimates the former based on the sample. If so, the question comes down to why one may replace $P(\theta,D)$ by $P(\theta|D)$ when everything is conditioned on $D$. –  whuber Oct 10 '12 at 19:52
@whuber. No, I didn't get what you mean by p(x|D) is an expectation, it is probability isn't it. Expectation is something that you get when you multiply f(x)*p(x). I am just a beginner. So can you please provide some clarifications? –  user34790 Oct 10 '12 at 19:57
–  whuber Oct 10 '12 at 19:58
@Whuber. I do get what expectation and probability is but my question is how come p(x|D) is expectation in the above case it is probability isn't it? –  user34790 Oct 10 '12 at 20:06
The trick is to do pattern matching: Writing $g(\theta)=P(x|\theta,D)$ and $f(\theta)=P(\theta,D)$, the definition of $p(x|D)$ becomes $\int g(\theta) f(\theta) d\ \theta$. Now using the letter "$x$" instead of $\theta$ gives you precisely Wikipedia's definition of $E(g(x))$. Reverting to the original meanings of $x$ and $\theta$, this is the expected value of $P(x|\theta,D)$--thought of as a function of $\theta$--when the probability law for $\theta$ is given by the density $P(\theta,D)$. –  whuber Oct 10 '12 at 20:13

May I use a slightly different (but still sloppy, confusing random objects with their realizations, etc.) notation? I didn't watch the lecture, but it seems that you know the value of $n$ observations $(x_1,\dots,x_n)=x^{(n)}$, and you are interested in the predictive density $f(x_{n+1}\mid x^{(n)})$. Marginalizing, using the product rule, and using the fact that $x_1,\dots,x_n,x_{n+1}$ are conditionally iid given $\theta$, we have $$f(x_{n+1}\mid x^{(n)}) = \int f(x_{n+1},\theta\mid x^{(n)})\,d\theta = \int f(x_{n+1}\mid\theta,x^{(n)})\,\pi(\theta\mid x^{(n)})\,d\theta$$ $$= \int f(x_{n+1}\mid\theta)\,\pi(\theta\mid x^{(n)})\,d\theta = (*)$$ Please, be sure that you understand the three equalities above. The last integral is the integral of "stuff" times a density (the posterior), so it is the expectation of "stuff" $$(*) = \mathbb{E}\left[f(x_{n+1}\mid\Phi)\right] \, ,$$ where the distribution of $\Phi$ has density $\pi(\,\cdot\mid x^{(n)})$.
Suppose that you have a sequence of iid random variables $\Phi_1,\Phi_2,\dots$, such that $\Phi_i\sim\Phi$. By the strong law of large numbers $$\frac{1}{N} \sum_{i=1}^N f(x_{n+1}\mid\Phi_i) \to \mathbb{E}\left[f(x_{n+1}\mid\Phi)\right] \, ,$$ almost surely, as $N\to\infty$.
Actually, if you are doing MCMC, your sequence of $\Phi_i$'s (a Markov chain) will be a dependent sequence, but given some regularity conditions that the chain satisfies, the Ergodic Theorem guarantees that you have the almost sure convergence described above.
(+1) @zen. I believe $N$ is the size of the $i.i.d.$ sample of $(\Phi_i)_{1\leq i\leq N}$ and the $a.s.$ convergence is indeed for $N \rightarrow \infty$ with fixed $n$. This is just to mention that you do not have a convergence theorem for (*) as $n \rightarrow \infty$ in the general case...Therefore the reader should really differenciate what pertains to $n$ (the number of measurements) and what pertains to $N$ (the sample size of simulated data from the posterior distribution). –  julien stirnemann Oct 10 '12 at 21:19
Yes. I'm taking $N\to\infty$ for some fixed $n$. Did you understand it differently? –  Zen Oct 10 '12 at 22:12
No, all is good. just: $N$ (capital n) is not defined and the fact that you have 2 "n" notations (caps and small) with 2 potential levels of convergence --> could be misinterpreted... –  julien stirnemann Oct 10 '12 at 23:37