# MLE Derivation for AR Model

So I am trying to derive the MLE for an AR(1) model. Here are my thoughts thus far:

The AR process is: $$z_t = \delta + \psi_1z_{t-1} + \epsilon_t$$

The expected value of $$z_t = \frac{\delta}{1 - \psi_1}$$.

The variance of $$z_t = \frac{1}{1 - \psi_1^2}$$.

So this is where I am getting caught up.

I have the PDF of $$z_t$$ as:

\begin{align} f(z_t;\theta) &= (2 \pi \sigma^2)^{-\frac{1}{2}} \exp \left [-\frac{1}{2} \left (\frac{z_t - \mathbb{E}[z_t]}{\sqrt{\sigma^2}}\right ) \right] \\ &= \left (2 \pi \frac{1}{1 - \psi_1^2} \right )^{-\frac{1}{2}} \exp \left [-\frac{1}{2} \left (\frac{z_t - \frac{\delta}{1 - \psi_1}} {\sqrt{\frac{1}{1 - \psi_1^2}}} \right )^2 \right] \\ &= \left (2 \pi \frac{1}{1 - \psi_1^2} \right )^{-\frac{1}{2}} \exp \left [-\frac{1}{2} \left (\frac{ \left(z_t - \frac{\delta}{1 - \psi_1} \right )^2}{\frac{1}{1 - \psi_1^2}} \right ) \right] \\ &= \left (2 \pi \frac{1}{1 - \psi_1^2} \right )^{-\frac{1}{2}} \exp \left [-\frac{1 - \psi_1^2}{2} \left( z_t - \frac{\delta}{1 - \psi_1} \right)^2 \right] \end{align}

Now, can I assume i.i.d. here? I feel like no because then I would have a time series that is just white noise right? However, if I did assume i.i.d., I would have:

$$\mathscr{L} = \prod_{t=1}^T \left (2 \pi \frac{1}{1 - \psi_1^2} \right )^{-\frac{1}{2}} \exp \left [-\frac{1 - \psi_1^2}{2} \left( z_t - \frac{\delta}{1 - \psi_1} \right)^2 \right]$$

And then from here what exactly would my log likelihood function be? I feel like I am totally screwing this up but this is what I have for it:

$$\ln \mathscr{L} = -\frac{T}{2} \ln \left ( 2 \pi \frac{1}{1 - \psi_1^2} \right ) - \frac{(1 - \psi_1^2) \sum_{t=1}^T \left (z_t - \frac{\delta}{1 - \psi_1} \right )^2}{2}$$

Any help is greatly appreciated! Thank you!!

• What is your $\theta$? – StubbornAtom Feb 27 at 6:22
• @StubbornAtom Was this meant to be a comment on my answer (I don't see a $\theta$ in the question)? I use $\theta$ conventionally to represent the vector of parameters ($\theta = (\delta, \psi_1, \sigma^2)$). – Chris Haug Feb 27 at 13:17
• @ChrisHaug I do see a $\theta$ in the question. OP in his comment to you says that $\epsilon_t$'s are standard normal, so I suppose $\theta = (\delta, \psi_1)$. – StubbornAtom Feb 27 at 13:46
• You're right, my bad. But whether $\sigma^2 = 1$ or not really doesn't matter here. – Chris Haug Feb 27 at 14:13

## 2 Answers

No, the $$z_t$$ are not independent, so that doesn't work.

The likelihood is a joint probability, so start there rather than from a single observation:

$$\mathcal{L}(\theta) = p(z_1, ..., z_T| \theta)$$

The recursive definition of the process (the first equation you show) gives you the transition distribution directly:

$$z_t | (z_1,...,z_{t-1}, \theta) \sim \mathcal{N}(\delta + \psi_1 z_{t-1}, \sigma^2)$$

So, it would be ideal if we could express our joint probability in terms of these transition distributions, then we can just plug them in. You can recursively condition on the previous observations to decompose the joint distribution like this:

\begin{align} \mathcal{L}(\theta) &= p(z_1, ..., z_T| \theta) \\ &= p(z_T|z_1,...,z_{T-1};\theta) \cdot p(z_1,...,z_{T-1}|\theta)\\ &= p(z_T|z_1,...,z_{T-1};\theta) \cdot p(z_{T-1}|z_1,...,z_{T-2};\theta) \cdot p(z_1,...,z_{T-2}|\theta)\\ &= \quad... \\ &= p(z_T|z_1,...,z_{T-1};\theta) \cdot p(z_{T-1}|z_1,...,z_{T-2};\theta) \cdot ... \cdot p(z_1|\theta) \end{align}

Plug in the normal transition distributions from above and you're (almost) done. There's a small issue: what is $$p(z_1|\theta)$$? Many definitions of AR(1) processes do not actually make this explicit. Typically, what is implied is that it has the "stationary distribution":

$$z_1 | \theta \sim \mathcal{N}\left(\frac{\delta}{1-\psi_1},\frac{\sigma^2}{1-\psi_1^2}\right)$$

• Actually no; that is not correct. I forget to specify the distribution of $\epsilon_t$ so my bad on that. $\epsilon_t$ follows a standard normal distribution. Hence, $z_t$ has a mean of $\frac{\delta}{1 - \psi_1}$ and variance of $\frac{1}{1 - \psi_1^2}$. I am not sure if that is what you were asking, but hopefully that helps – James Bradley Feb 27 at 1:23
• @JamesBradley If you don't specify the distribution of $z_1$, you actually can't derive the marginal distribution of $z_t$ just from the transition distribution, without making additional assumptions. So no, it is not true that, based on the definition given, $z_t$ has mean $\frac{\delta}{1-\psi_1}$ and variance $\frac{1}{1-\psi_1^2}$. It is true if you set the distribution of $z_1$ to the stationary distribution as described above, but this is an independent assumption. – Chris Haug Feb 27 at 13:14

I'm not directly answering your question, but a quick note on the construction of the likelihood function in your model. The likelihood of a $$T$$-sized sample of $$\mathbf{e} = \left[ \epsilon_{1}, \, \epsilon_{2}, \ldots, \, \epsilon_{T} \right]^{\mathsf{T}}$$ of i.i.d. normal distributed $$\epsilon \sim N(0,\sigma^{2})$$ is $$L(\mathbf{e}) = (2 \pi \sigma^{2})^{\frac{T}{2}} \exp \left[ \frac{- \mathbf{e}^\mathsf{T} \mathbf{e}}{2\sigma^{2}} \right]$$ But obviously you observe $$\mathbf{z} = \left[ z_{1}, \, z_{2}, \ldots, \, z_{T} \right]^{\mathsf{T}}$$ instead of $$\mathbf{e}$$. From your AR(1) setup you have $$\mathbf{e} = \mathbf{G} \mathbf{z} - \delta \mathbf{I}_{T}$$ where $$\mathbf{G} = \begin{bmatrix} \sqrt{1-\psi_{1}^2} & 0 & 0 & \cdots & 0 \\ -\psi_{1} & 1 & 0 & \cdots & 0 \\ 0 & -\psi_{1} & 1 & \cdots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \cdots & 1 \end{bmatrix},$$ (see Prais & Winsten, eqn. 15). Then $$L(\mathbf{z}) = L(\mathbf{e}) \left| \frac{d \mathbf{e}}{d\mathbf{z}} \right|$$ where $$\left| \frac{d \mathbf{e}}{d\mathbf{z}} \right|$$ is the Jacobian (determiant) of the transformation, in this case $$\operatorname{det} \mathbf{G}$$, which works out to be $$\left| 1 -\psi_{1}^{2} \right|^{\frac{1}{2}}$$ because of all the off-diagonal zeros in $$\mathbf{G}$$. See Beach & MacKinnon (1978) for more details.