Difference between stationarity and ergodicity I am currently studying time series. However, I have trouble understanding how ergodicity and stationarity differ.
Could someone clarify the difference between the two concepts in simple terms and maybe give an example?
 A: A very simple example of a series which is stationary but not ergodic is the following:
$$ X_t = \sin(2\pi t + \Omega), \quad t=0,2,3,\dotsc  $$
where $\Omega \sim \mathcal{Unif}(0, 2\pi)$. Observe that by the periodicity of $\sin$ we have
$$ X_0=\sin(\Omega), X_1=\sin(\Omega), X_2=\sin(\Omega), \dotsc  $$ so the time mean is $\sin(\Omega)$, a random variable, different from the ensemble mean of $0$.
A: Adapted from an answer of mine on dsp.SE, with editing to suit the sensitivities of  stats.SE readers....
A random process is a collection of random variables, one for each time instant under consideration.  Typically this may be continuous time ($-\infty < t < \infty$) or discrete time (all integers $n$, or all time instants $nT$ where $T$ is called the sampling interval by denizens of dsp.SE.  Here, since the question is about time series, we will restrict attention to the case when $t$ can only take on integer values, and denote the $n$-th random variable as $X_n$

*

*Stationarity refers to the distributions of the random variables. Specifically, in a stationary process, all the random variables have the same distribution function, and more generally, for every positive integer $M$ and $M$ integer-valued time instants $n_1, n_2, \ldots, n_M$, the joint distribution of the $N$ random variables $X_{n_1}, X_{n_2}, \cdots, X_{n_M})$ is the same as the joint distribution of $X_{n_1+k}, X_{n_2+k}, \cdots, X_{n_M+k})$.  That is, if we shift all time instants by $\tau$, the statistical description of the process does not change at all: the process is stationary.

Stationarity usually has a different meaning for people interested in time series, machine learning, econometrics etc where the above notion is referred to as strict stationarity (if it is ever mentioned at all). For these folks, stationarity usually means what other people call weak stationarity or wide-sense-stationarity: meaning that all the random variables have the same mean, and the autocorrelation function $E[X_nX_m]$ depends only on the difference $m-n$ in the positions of the the two random variables on the time axis, and not upon $m$ and $n$ individually.

*Ergodicity, on the other hand, doesn't look at statistical properties of the random variables but at the sample paths, i.e. what you observe physically. Referring back to the random variables, recall that random variables are mappings from a sample space to the real numbers; each outcome is mapped onto a real number, and different random variables will typically map any given outcome to different numbers.  So, imagine that some higher being as performed the experiment which has resulted in an outcome $\omega$ in the sample space, and this outcome has been mapped onto (typically different) real numbers by all the random variables in the process: specifically, the random variable $X_n$ has mapped $\omega$ to a real number we shall denote as $x_n$: $X_n(\omega) = x_n$. The sequence
$$\cdots,~ x_{-n}, ~\cdots, ~ x_{-2},~ x_{-1},~ x_0,~ x_1,~ x_2,~ \cdots, ~x_n, ~\cdots$$ is the sample path corresponding to outcome $\omega$ in the sample space. Different outcomes $\omega^\prime$ will give us different sample paths. Ergodicity then deals with properties of the sample paths and how these properties relate to the statistical properties of the random variables comprising the stationary random process.

For example, one expects that the empirical CDF of a large number of points on a given sample path should resemble the actual (common) CDF shared by all the random variables $X_n$, and the empirical CDF should converge to the common CDF as the number of points increases without bound. The process is called ergodic if such convergence is assured, if not for every possible sample path, at least for almost all possible sample paths.

Now, for a sample path from a (strictly) stationary process, we can compute the time average of the sample path (sample mean in stats.SE parlance)
$$\bar{x} = \frac{1}{2N+1} \sum_{i=-N}^N x_i$$  but, what does $\bar{x}$ have to do with $\mu = E[X_n]$, the common mean of all the random variables in the the random process? (Note that it doesn't matter which value of $n$ we use; all the random variables have the same distribution and so have the same mean (assuming the mean exists)).  If the average value of a sample path converges to the mean value of the process when the sample path is observed long enough, we say that the process is ergodic or more specifically, mean-ergodic.  That is, ergodicity is what enables us to connect the results of the two calculations and to assert that
$$\lim_{N\to \infty}\bar{x} = \lim_{N\to \infty}\frac{1}{2N+1} \sum_{-N}^N x_n ~~~
\textbf{equals}  ~~~\mu = E[X_n] = \int_{-\infty}^\infty xf_X(x) \,\mathrm dx$$ where $f_X(x)$ is the common pdf of all the $X_n$'s. (If the $X_n$'s are discrete random variables, replace the integral with the appropriate sum). As noted before, a process for which such equality holds is said to be mean-ergodic, and it is known that a process is mean-ergodic if its autocovariance function $C_X(n)$ has the property:
$$\lim_{N\to\infty}\frac{1}{2N+1}\sum_{i=-N}^N C_X(i) = 0.$$
Thus, not all stationary processes need be mean-ergodic. But there are other forms of ergodicity too. For example, for an autocovariance-ergodic process, the autocovariance function of a finite segment (say for $n\in [-N, N]$ of the sample path converges to the autocovariance function $C_X(\tau)$ of the process as $N\to \infty$. A blanket statement that a process is ergodic might mean any of the various forms or it might mean a specific form; one just can't tell,
As an example of the difference between the two concepts, suppose that $X(t) = Y$ for all $t$ under consideration.  Here $Y$ is a random variable. This is a (strictly) stationary process: each $X(t)$ has the same distribution (namely, the distribution of $Y$), same mean
$E[X(t)] = E[Y]$, same variance etc.;
each $X(t_1)$ and $X(t_2)$ have the same
joint distribution (though it is degenerate) and so on.  But the process is not
ergodic because each sample path is a constant.  Specifically, if a trial of the experiment
(as performed by you, or by a superior being) results in $Y$ having
value $\alpha$, then the sample path of the random process that corresponds
to this experimental outcome has value $\alpha$ for all $n$, and so the sample mean of any number of samples is $\alpha$,no matter how long you observe this (rather boring) sample path. It follows that the limiting value of the sample mean is $\alpha$ too, and not converging to $E[Y]$ at all.  In a parallel universe,
the trial would result in $Y = \beta$ and the sample path in that
universe would have value $\beta$ for all $n$. In short, any inference about the statistics of the random process from observation of a sample path of this process is way off-base.
It is not easy to write mathematical specifications to
exclude such trivialities from the class of stationary processes, and so this is a very minimal example of a (strictly) stationary random process that is not ergodic.  Note also that this example is essentially the same as the one in the answer by kjetil b halvorsen.
Can there be a random process that is not stationary but is ergodic? Well, NO, not if by ergodic we mean ergodic in every possible way one can think of: for example, if we measure the fraction of time during which a long segment of the sample path $x(t)$ has value at most $\alpha$, this is a good estimate of $P(X(t) \leq \alpha) = F_X(\alpha)$, the value of the (common) CDF $F_X$ of the $X(t)$'s at $\alpha$ if the process is assumed to be ergodic with respect to the distribution  functions.  But, we can have random processes that are not strictly stationary but are nonetheless mean-ergodic and autocovariance-ergodic.  For example, consider the process
$\{X_n\colon X_n= \cos (n + \Theta), -\infty < n < \infty\}$
where $\Theta$ takes on four equally likely values $0, \pi/2, \pi$ and $3\pi/2$.
Note that each $X_n$ is a discrete random variable that takes on four equally likely values $\cos(n), \cos(n+\pi/2)=-\sin(n), \cos(n+\pi) = -\cos(n)$ and $\cos(n+3\pi/2)=\sin(n)$,  Thus, $X_n$ and $X_m$ have different distributions, and so the process is not stationary. On the other hand,
$$E[X_n] = \frac 14\cos(n)+
\frac 14(-\sin(n)) + \frac 14(-\cos(n))+\frac 14 \sin(n)  = 0$$ for every $n$ while
\begin{align}
E[X_nX_m]&= \left.\left.\frac 14\right[\cos(n)\cos(m) +  (-\cos(n))(-\cos(m)) + \sin(n)\sin(m) + (-\sin(n))(-\sin(m))\right]\\
&= \left.\left.\frac 12\right[\cos(n)\cos(m) + \sin(n)\sin(m)\right]\\
&= \frac 12 \cos(m-n).
\end{align}
In short, the process has zero mean and its autocorrelation (and autocovariance) function depends only on the time difference $m-n$, and so the process is wide-sense-stationary . But it is not strictly stationary (or even first-order stationary for those who like to think of such details). Now, when the experiment is performed and the value of $\Theta$ is known, we get the sample path which clearly must be one of $\pm \{\cos(n)\}$ and $\pm \{\sin(n)\}$ for all of which the sample mean converges to $0$ which is the mean of the process. Similarly, the autocorrelation function of the sample path converges to $\frac 12 \cos(\tau)$, same as the autocorrelation function of the process. Thus, this process is mean-ergodic and autocorrelation-ergodic even though it is not strictly stationary at all. Note that the process is  not ergodic with respect to the distribution function, that is, it cannot be said to be ergodic in all respects.
Can a process be ergodic but not stationary in any sense, not even the wide sense used in the previous paragraph? Well, if even the mean ot the process is time-varying, to which number should $\frac{1}{2N+1} \sum_{i=-N}^N x_i $ converge as $N\to\infty$ so that we can say that the process is mean-ergodic? For a process to be ergodic, something must be fixed to which convergence can occur.
