Conceptual questions: Variance of a process

Question

Wikepedia, at Variance of Autoregressive model, mentions an expression of variance for an AR(1) process. I am learning signal processing (beginner level) and facing difficulty in understanding some basic relations. It shall be really helpful if the following doubts are answered.

An AR(1) process is defined by $$X_t = c+\theta X_{t-1} + \epsilon_t$$

where $\epsilon_t$ is a zero mean white Gaussian noise. We may compute expected values

$$E[X_t] = E[c] + \theta E[X_{t-1}]+E[\epsilon_t]$$

implying $E[X_t] = \mu = c/(1-\theta)$.

I am interested in the variance, which is defined as $$\text{Var}(X_t) = E[X_t^2] - \mu^2.$$ However, I read that $$\text{Var}(X_t) = \sigma_\epsilon^2/(1-\theta^2).$$

How is this derived?

Answer 1

Compute the variance using the only information you have--the definition of the process itself--noting that adding a constant $c$ to any random variable does not change its variance:

$$\text{Var}(X_t) = \text{Var}(c + \theta X_{t-1} + \epsilon_t) = \text{Var}(\theta X_{t-1}) + \text{Var}(\epsilon_t) + 2\text{Cov}(\theta X_{t-1}, \epsilon_t).$$

Let's take the three terms at the end one at a time, from left to right. We can factor the $\theta$ out of $\text{Var}(\theta X_{t-1})$, where it must appear as $\theta^2$ (because variances are quadratic forms). I presume $\sigma_\epsilon^2$ is your name for $\text{Var}(\epsilon_t)$, so I will use it in the next expression. Now you probably were asked to assume that $\epsilon_t$ and $X_{t-1}$ are independent, or at least uncorrelated. In either of those cases the covariance term drops out because it is zero.

These considerations lead to the simplified formula

$$\text{Var}(X_t) = \theta^2\, \text{Var}(X_{t-1}) + \sigma_\epsilon^2 + 0 = \theta^2\, \text{Var}(X_{t-1}) + \sigma_\epsilon^2.$$

Finally, at some point you were probably invited to assume the process is second order stationary. That implies, inter alia, that

$$\text{Var}(X_t) = \text{Var}(X_{t-1}).$$

Plug this into the preceding equation to eliminate $\text{Var}(X_{t-1})$ and solve for $\text{Var}(X_t)$.