State space representation of ARMA(p,q) from Hamilton

I have been reading Hamilton Chapter 13 and he has the following state space representation for an ARMA(p,q). Let $r = \max(p,q+1)$.Then the ARMA (p,q) process is as follows: \begin{aligned} y_t -\mu &= \phi_1(y_{t-1} -\mu) + \phi_2(y_{t-2} -\mu) + ... + \phi_3(y_{t-3} -\mu) \\ &+ \epsilon_t + \theta_1\epsilon_{t-1} + ... + \theta_{r-1}\epsilon_{t-r+1}. \end{aligned}
Then, he defines the State Equation as follows:

$$\xi_{t+1} = \begin{bmatrix} \phi_1 & \phi_2 & \dots & \phi_{r-1} & \phi_r \\ 1 & 0 & \dots & 0 & 0 \\ \vdots & \vdots & \dots & 0 & 0 \\ 0 & 0 & \dots &1 &0 \end{bmatrix} \xi_t + \begin{bmatrix} \epsilon_{t+1} \\ 0\\\vdots \\ 0 \end{bmatrix}$$

and the observation equation as:

$$y_t = \mu + \begin{bmatrix} 1 & \theta_1 & \theta_2 & \dots & \theta_{r-1} \\ \end{bmatrix} \xi_t.$$

I do not understand what the $\xi_t$ is in this case. Because in his AR(p) representation it is $\begin{bmatrix} y_{t} - \mu \\ y_{t-1} - \mu\\\vdots \\ y_{t-p+1} - \mu \end{bmatrix}$ and in his MA (1) representation it is $\begin{bmatrix} \epsilon_{t} \\ \epsilon_{t-1} \end{bmatrix}$.

Could someone explain this to me a little bit better?

Hamilton shows that this is a correct representation in the book, but the approach may seem a bit counterintuitive. Let me therefore first give a high-level answer that motivates his modeling choice and then elaborate a bit on his derivation.

Motivation:

As should become clear from reading Chapter 13, there are many ways to write a dynamic model in state space form. We should therefore ask why Hamilton chose this particular representation. The reason is that that this representation keeps the dimensionality of the state vector low. Intuitively, you would think (or at least I would) that the state vector for an ARMA($p$,$q$) needs to be at least of dimension $p+q$. After all, just from observing say $y_{t-1}$, we cannot infer the value of $\epsilon_{t-1}$. Yet he shows that we can define the state-space representation in a clever way that leaves the state vector of dimension of at most $r = \max\{p, q + 1 \}$. Keeping the state dimensionality low may be important for the computational implementation, I guess. It turns out that his state-space representation also offers a nice interpretation of an ARMA process: the unobserved state is an AR($p$), while the MA($q$) part arises due to measurement error.

Derivation:

Now for the derivation. First note that, using lag operator notation, the ARMA(p,q) is defined as: $$(1-\phi_1L - \ldots - \phi_rL^r)(y_t - \mu) =(1 + \theta_1L + \ldots + \theta_{r-1}L^{r-1})\epsilon_t$$ where we let $\phi_j = 0$ for $j>p$, and $\theta_j = 0$ for $j>q$ and we omit $\theta_r$ since $r$ is at least $q+1$. So all we need to show is that his state and observation equations imply the equation above. Let the state vector be $$\mathbf{\xi}_t = \{\xi_{1,t}, \xi_{2,t},\ldots,\xi_{r,t}\}^\top$$ Now look at the state equation. You can check that equations $2$ to $r$ simply move the entries $\xi_{i,t}$ to $\xi_{i-1,t+1}$ one period ahead and discard $\xi_{r,t}$ in the state vector at $t+1$. The first equation, defining $\xi_{i,t+1}$ is therefore the relevant one. Writing it out: $$\xi_{1,t+1} = \phi_1 \xi_{1,t} + \phi_2 \xi_{2,t} + \ldots + \phi_r \xi_{r,t} + \epsilon_{t+1}$$ Since the second element of $\mathbf{\xi_{t}}$ is the first element of $\mathbf{\xi_{t-1}}$ and the third element of the $\mathbf{\xi_{t}}$ is the first element of $\mathbf{\xi_{t-2}}$ and so on, we can rewrite this, using lag operator notation and moving the lag polynomial to the left hand side (equation 13.1.24 in H.): $$(1-\phi_1L - \ldots - \phi_rL^r)\xi_{1,t+1} = \epsilon_{t+1}$$ So the hidden state follows an autoregressive process. Similarly, the observation equation is $$y_t = \mu + \xi_{1,t} + \theta_1\xi_{2,t} + \ldots + \theta_{r-1}\xi_{r-1,t}$$ or $$y_t - \mu = (1 + \theta_1L + \ldots + \theta_{r-1}L^{r-1})\xi_{1,t}$$ This does not look much like an ARMA so far, but now comes the nice part: multiply the last equation by $(1-\phi_1L - \ldots - \phi_rL^r)$: $$(1-\phi_1L - \ldots - \phi_rL^r)(y_t - \mu) = (1 + \theta_1L + \ldots + \theta_{r-1}L^{r-1})(1-\phi_1L - \ldots - \phi_rL^r)y_t$$ But from the state equation (lagged by one period), we have $(1-\phi_1L - \ldots - \phi_rL^r)\xi_{1,t} = \epsilon_{t}$! So the above is equivalent to $$(1-\phi_1L - \ldots - \phi_rL^r)(y_t - \mu) = (1 + \theta_1L + \ldots + \theta_{r-1}L^{r-1})\epsilon_{t}$$ which is exactly what we needed to show! So the state-observation system correctly represents the ARMA(p,q). I was really just paraphrasing Hamilton, but I hope that this is useful anyway.

• I'm not totally sold on the interpretation of the state, though. When you write the first line of the state transition equation, it seems like an equation that conflicts with the assumed model. Also I find it strange that you assume observed data is at the same time hidden/latent. Feb 11 '17 at 0:32
• You are right, the state is indeed not the same as $y_t$. Thanks for pointing this out. I corrected it, should be fine now. Btw, in general we could have observed variables in the state vector, see for instance the AR(p) example. There the hidden variable can be thought of as next period's value, $y_{t+1}$. Feb 11 '17 at 1:12
• Thank you! But I al still confused as to what $\xi$ is in this state space representation. Not for example his definition of $\xi$ in equation 13.1.15 and 13.1.14 for and AR(p) and MA(1) process.My confusion is, if I put this in matlab, what numbers am i getting in $\xi$? Feb 11 '17 at 14:45
• What is confusing here is that state space modelling is concerned with a hidden state, while with ARMA processes we do not think of variables as hidden. The state space representation and the (Kalman) Filtering techniques are motivated by filtering out the unobserved state. For ARMA processes, we just use the formulation of state-space models so we can estimate the parameters using the Kalman Filter. So we somewhat arbitrarily define the hidden state in 13.1.4 as next period's observation $y_{t+1}$ while in 13.1.22, the state is a new variable that does not appear in the original model. Feb 11 '17 at 16:53
• To answer your question about Matlab: if you start off from an ARMA(p,q), the $\xi$ is not a variable that appears in that model. However, the state space representation actually offers a different interpretation of the ARMA(p,q): the hidden state could be the variable that you are interested in and the MA(q) structure arises because of measurement error. You can write down an AR(1) and add some white noise to see that an ARMA structure arises. Feb 11 '17 at 17:02

This is the same as above, but I thought I would provide a shorter, more concise answer. Again, this is Hamilton's representation for a causal ARMA($p$,$q$) process, where $r=\max(p,q+1)$. This $r$ number will be the dimension of the state vector $(\xi_t, \xi_{t-1},\ldots, \xi_{t-r+1})'$, and it is needed to make the number of rows of the state match up with the number of columns of the observation matrix. That means we also have to set coefficients to zero whenever the index is too big.

1. Observation Equation

\begin{align*} &\phi(B)(y_t - \mu) = \theta(B)\epsilon_t \\ \iff &(y_t - \mu) = \phi^{-1}(B)\theta(B)\epsilon_t \tag{causality}\\ \iff &y_t = \mu + \phi^{-1}(B)\theta(B)\epsilon_t \\ \iff &y_t = \mu + \theta(B)\phi^{-1}(B)\epsilon_t \\ \iff &y_t = \mu + \theta(B)\xi_t \tag{letting $\xi_t = \phi^{-1}(B)\epsilon_t$}\\ \iff &y_t = \mu + \left[\begin{array}{ccccc}1 & \theta_1 & \theta_2 & \ldots & \theta_{r-1} \end{array}\right] \underbrace{\left[\begin{array}{c} \xi_t \\ \xi_{t-1} \\ \vdots \\ \xi_{t-r+1} \end{array}\right]}_{\text{the state vector}} + 0\tag{this is where we need $r$}. \end{align*}

1. State Equation

\begin{align*} &\xi_t = \phi^{-1}(B)\epsilon_t \\ \iff &\phi(B)\xi_t = \epsilon_t \\ \iff &(1 - \phi_1 B - \cdots - \phi_rB^r) \xi_t = \epsilon_t \\ \iff &\xi_t = \phi_1 \xi_{t-1} + \cdots + \phi_r \xi_{t-r} + \epsilon_t \\ \iff & \left[\begin{array}{c} \xi_t \\ \xi_{t-1} \\ \xi_{t-2} \\ \vdots \\ \xi_{t-r+1} \end{array}\right] = \left[ \begin{array}{ccccc} \phi_1 & \phi_2 & \phi_3 & \cdots & \phi_r \\ 1 & 0 & 0 & \cdots & 0 \\ 0 & 1 & 0 & \cdots & 0 \\ \vdots & \vdots & \ddots & \cdots & 0 \\ 0 & 0 &\cdots & 1 & 0 \end{array}\right] \left[ \begin{array}{c} \xi_{t-1} \\ \xi_{t-2} \\ \vdots \\ \xi_{t-r} \end{array}\right] + \left[ \begin{array}{c} \epsilon_t \\ 0 \\ \vdots \\ 0 \end{array}\right]. \end{align*}

• This makes it finally clear where those state equations come from. I think stating it like that is didactically much better than just giving those random appearing equations with the note that it turns out right.
– Alex
Jul 28 '18 at 10:47
• @CowboyTrader yes, that's right. At least for this ARMA representation. There are some others. Jul 9 '19 at 18:36
• @CowboyTrader no, but I would say this is a sensible feeling because literature on state space models is biased towards filtering. Recursive prediction equations exist for linear Gaussian state space models exist, but you do get the filtering stuff as an added bonus. Jul 9 '19 at 19:21
• @CowboyTrader feel free to send me an email. I know not everyone loves extended discussions in comments, so it might be easier to do that. Jul 9 '19 at 20:13
• I see that it is proved, but, could you please help give some intuition? What are the state variables, what is the t=0 state vector? Jul 17 '19 at 0:16