Derivation of sample autocovariance

Question

The autocovariance is defined as

$$\gamma(t,s) = Cov(X_{t}, X_{s})=E[(X_{t}-\mu_{t})(X_{s}-\mu_{s})]$$

When we have a stationary process the only thing that matters is the lag between the variables:

$$\gamma_{k} = Cov(X_{t}, X_{t-k})=E[(X_{t}-\mu)(X_{t-k}-\mu)]$$

However, the expectation means that we are summing over all possible values of the random variable $X$. For example:

$$\gamma_{k}=\int\cdot\cdot\cdot\int(X_{t}-\mu)(X_{t-k}-\mu)f(X_{t},...,X_{t-k})dX_{t},...,dX_{t-k}$$

Given this definition of $\gamma_{k}$, how can we derive the sample autocovariance:

$$\gamma_{k} = \frac{1}{N}\sum_{t=0}^{N}(X_{t}-\mu)(X_{t-k}-\mu)$$

in which we are summing over $t$ instead of $X$. I think this is related to an ergodicity assuption, but I haven't found a very detailed explanation about it.

Another question: Is this definition of sample autocovariance correct only when the process is stationary?

Answer 1

1) If a discrete-time stochastic process $\{X_t\}$ is ergodic, then any continuous function of it is also ergodic. Consider the following continuous function of an ergodic stochastic process: $$f\left(\{X_t\}\right) = (X_{t}-\mu)(X_{t-k}-\mu) $$

which is also ergodic (the function is continuous, for each $t$, because it represents a transformation of $\{X_t\}$ by the application of the lag operator which is a continuous transformation, and by using multiplication and addition).

2) By the Birkhoff-Kinchin theorem, if a process is ergodic, and its first moment (expected value) exists (which is then constant due to ergodicity), then its mean over time tends almost surely to its expected value (the "ensemble" mean).

3) The autocovariance of the process $\{X_t\}$ is

$$\gamma_{k} = Cov(X_{t}, X_{t-k})=E[(X_{t}-\mu)(X_{t-k}-\mu)] = E\left[f\left(\{X_t\}\right)\right] $$

i.e. it equals the ensemble mean of the ergodic process $f\left(\{X_t\}\right)$. Since this process too is ergodic, its mean over time, which is

$$\hat \gamma_{k} = \frac{1}{N}\sum_{t=0}^{N}(X_{t}-\mu)(X_{t-k}-\mu)$$

tends almost surely to its ensemble mean, which is the theoretical covariance. In other words, $\hat \gamma_{k}$ is a strongly consistent estimator of $\gamma_{k}$.