I am reading the 2004 MCMC paper by Roberts and Rosenthal and was hoping someone could shed some further light on Example 3. I restate Theorem 4 first, which the example relies on.
Theorem 4. If a Markov chain on a state space with countably generated $\sigma$-algebra is $\phi$-irreducible and aperiodic, and has a stationary distribution $\pi$, then for $\pi$-a.e. $x\in \mathcal{X}$, $$ \lim_{n\to\infty}||P^n(x, \cdot)-\pi(\cdot)||=0. $$ In particular, $\lim_{n\to\infty}P^n(x, A)=\pi(A)$ for all measurable $A\subseteq \mathcal{X}$.
Now for the example (due to Charles Geyer):
Example 3. Let $\mathcal{X}=\{1, 2, \dots\}.$ Let $P(1, \{1\})=1$, and for $x\geq 2$, $P(x, \{1\})=1/x^2$ and $P(x, \{x+1\})=1-1/x^2$. Then chain has stationary distribution $\pi(\cdot)=\delta_1(\cdot)$ and it is $\pi$-irreducible and aperiodic. On the other hand, if $X_0=x\geq 2$, then $\mathbf{P}[X_n=x+n \text{ for all }n]=\prod_{j=x}^\infty(1-1/j^2)>0$ so that $||P^n(x, \cdot)-\pi(\cdot)||/{\to}0$. Here Theorem 4 holds for $x=1$ which is indeed $\pi$-a.e. $x\in\mathcal{X}$, but it does not hold for $x\geq 2$.
(In the example, I used $/\to$ to denote "does not converge to"; $||\cdot||$ denotes total variation distance.)
It is the final sentence of the example I cannot wrap my head around. Can someone explain the example differently?
After some more thought, I realized $\delta_1(\cdot)$ must be the Kronecker delta. Hence, $\pi(1)=1$ and $\pi(x)=0$ for $x\geq 2$ and the theorem holds if $x=1$ which happens with probability 1 (with respect to $\pi$). So the issue is simply that the theorem is for $\pi$-a.e., whereas elements with $\pi$-measure zero may have positive measure for finite $n$ (and in this particular case, the chain may keep on going). Is this (fairly vague) reasoning correct?