This is based on example 2.2 from Machine Learning: A Bayesian and Optimization Perspective by Theodoridis. Please note that I'm not at all familiar with ergodic theory and I'm reading this with background from Casella and Berger.

A wide-sense stationary process (WSS), by definition, is a stochastic process $(u_n)_{n \in \mathbb{Z}}$ such that for each $k$, $\mathbb{E}[u_k] = \mu$ for all $k$ (i.e., the means are identical) and if we define $$r(n, m) = \mathbb{E}[u_nu_m]$$ we have $r(n, n-k) = r(k)$ (i.e., the "autocorrelation" (what Theodoridis defines as such, technically the cross-product moment)) only depends on the difference between times.

Consider a WSS $(u_n)_{n \in \mathbb{Z}}$ for which the expected value is $\mu$ at each time, and $$\mathbb{E}[u_nu_{n-k}] = r_u(k)\text{.}$$ Let $v_n = au_n$, where $a$ is a random variable taking values in $\{0, 1\}$, each with probability $0.5$. Assuming independence of $a$ and $u_n$, we obtain $\mathbb{E}[v_n] = 0.5\mu$ and $\mathbb{E}[v_nv_{n-k}] = 0.5r_u(k)$.

Theodoridis then remarks:

Thus, $v_n$ is WSS. However, it is not covariance-ergodic. Indeed, some of the realizations will be equal to zero (when $a = 0$), and the mean value and autocorrelation, which will result from them as time averages, will be zero, which is different from the ensemble averages.

I don't understand any of the above quote, other than that $v_n$ is WSS. By "covariance-ergodic," the author talks about $$\text{cov}(k) = \lim_{N \to \infty}\dfrac{1}{2N+1}\sum_{n=-N}^{N}(u_n - \mu)(u_{n-k}-\mu)$$ with limits "in the mean-square sense" equaling $0$ (I think?).

Could someone enlighten me on this?


1 Answer 1


Let's leave aside the concept of ergodicity itself, which is rather deep, abstract and difficult, and use its consequences: when a stochastic process $\{X_i\}$ is ergodic, the "time" average (the average across the index, which often represents time, but this is not necessary) converges in probability to the ensemble average.
Ergodicity comes in levels: a process may be "ergodic for the mean", but not for the variance, etc.

A way to think about this is as a generalization of the Law of Large Numbers. LLN theorems have assumptions that make the sequences of random variables under examination ergodic (and here, the index does not necessarily represent time).

In most cases, "wide-sense" stationary processes over time (or more accurately "covariance-stationary" processes) are also ergodic, and so averaging over the available time-series observations provides a consistent estimator for the common mean (and then of the variance and of the covariance). But this is not always the case.

The author the OP mentions provides an example were we have stationarity but not ergodicity for the autocovariance, which implies that that the sample autocovariance in this case is not a consistent estimator of the true autocovariance.

I would suggest to look up Hamilton's Time Series Analysis, p. 46-47 for a discussion and another such example, and ch. 7 for some sufficient conditions that ensure that a stationary process is also ergodic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.