I am implementing a Gibbs sampler in order to estimate the parameters of a mixture model.

Assuming that the parameters are contained in a vector $\theta$ what I will do is:

  • Implement and run the sampler.
  • Wait to reach a stationary distribution (so to remove the impact of my initialization); say that we reach the stationary distribution at iteration $n$.
  • After iteration $n$, I start collecting the sampled parameters into a list $[\theta_n,...\theta_{n+m}]$
  • Last, I evaluate the expected value of $\theta$ doing $$ \theta_{m,estimated} = \sum_{i=1}^{m} \theta_{n+i}/m $$

Now, I have some questions:

  • Two subsequent samples $\theta_{n+i}, \theta_{n+i+1} $ will obviosly be correlated. Should I skip some samples in order to ensure independence between the samples I use to calculate my estimator? If yes, there is some standard heuristic to do this?
  • Can I say that (if the sampler is properly implemented) $\theta_{m,estimated}$ is a consistent estimator of $\theta$?
  • 4
    $\begingroup$ Your questions seem to be already answered on this site: for "skipping" samples (thinning) see this answer (thinning is not useful), and for conditions for consistency see this answer. Please look at those answers and re-post with more specific questions if any remain. $\endgroup$
    – EdM
    Jan 26 '19 at 21:44