If you can train your chain on data and can find out the transition matrix, you can predict. I don't see why prediction has to be related with ergodicity.

@David Assume the ideal world which we know from whose mixture the observation comes from. In this case, we know that only one of Gaussian in the mixture matters.

@youpilat13 MCMC doesn't have an algorithm to calculate the posterior distribution. It only generates the sample. After the sample is generated, you can do the inference on what the distribution of the sample looks like but it's a completely different story. The estimation process to figure out the distribution is not the MCMC sampling.

You may check the Wikipedia article(en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff‌​). It explains more clearly.

@Josh It's conditional on $x_0$ because we want to test our model with the observation ($x_0$, $y_0$). So the error is calculated with the label $y_0$. The expectation is taken over the training sets.

I don't know how you apply CLT here. CLT states that the distribution of the sum of random variables approaches to Gaussian.

By differencing, you don't explain the differenced part by model. You accepts the difference as the fact and don't explain it. And your model focuses on the explanation of the residual of differencing.

The sampler is to generate random samples and estimator is to estimate a quantity from the generated random sample. I'm not sure why you got confused with these two concepts. MCMC is not an estimator. It's to generate samples from the Bayes' rule. Same goes to the acceptance-rejection method.