Combining multiple parallel MCMC chains into one longer chain

Question

Let's say that one has run $m$ parallel MCMC chains where each chain has had burn-in. Let the resulting chains be denoted by $$ x_1^{(i)},\dots,x_N^{(i)} \quad \text{ for } i=1,\dots,m,$$ where $N$ is the length of each chain after burn-in.

If one would want to combine these chains into one long chain, is it as simple as concatenating them as $$x_1^{(1)},\dots,x_N^{(1)},\dots, x_1^{(m)},\dots, x_N^{(m)} ?$$

In my case, each $x_i$ is a parametervector $\theta_i$. My goal is to sample from the posterior $$ p(\theta \mid y),$$ where $y$ is the data. The reason for why I am interested in parallel chains is because they are needed for calculating potential scale reduction factor (PSRF).

Answer 1

Note that individual chains have serial dependence; values from separate chains don't, so if you wanted it to look like one long chain, just concatenating them wouldn't look right.

However, if you're just interested in the distribution, the order in the chain is irrelevant. You don't actually seek to concatenate the chains for that, you simply want to pool all the distributional information (treat them as one big sample). Certainly, if the chains are all converged to their stationary distribution, they'll all be samples from the same distribution -- you can combine those.

Indeed some people run a burn-in period and just draw a single value from many separate chains.

(Keeping the runs separate might help to judge if they actually have converged.)

If you're computing variance that accounts for the dependence structure, however, you would base it on the fact that the different runs are independent, but the values from within the same run are dependent.