# Auto regressive process, maximum likelihood estimator

A first-order autoregressive process, $X_0,\dots,X_n$, is given through the following conditional distributions: $X_i | X_{i-1},\dots,X_0 \sim \mathcal{N}(\alpha X_{i-1},1)$, for $i = 1,2,\dots,n$ and $X_0 \sim \mathcal{N}(0,1)$.

I know that the log-likelihood function $\ell{(\alpha)}$ is of the form: $\ell(\alpha) = - \frac{1}{2} \sum (x_i - \alpha x_{i-1})^2 + c$. But I don't know how to show that.

I found for $\hat{\alpha}_{ML}$ the following solution: $\hat{\alpha}_{ML} = \frac{s}{t}, \mathrm{where} \; s = \sum x_1 x_{i-1} \mathrm{and} \; t = \sum (x_{i-1})^2$. Is this right?

Then I have to show that this is the global maximum. If I take the second derivative I get a constant. Is this the sign that I got the global maximum, because the first derivative is linear wrt to $\alpha$? Right?

• The likelihood function of the sample is the joint density. Here the variables are not independent. Do you know how to apply the chain rule to decompose the joint density into conditional densities? Commented Oct 13, 2013 at 20:22
• I thought about that. But I don't get it... Do you have a more concrete hint? Commented Oct 14, 2013 at 5:51

Using the chain rule, the joint density here can be decomposed as (denoting $\mathbf X$ the collection of the $n+1$ random variables)

$$f_{\mathbf X}(x_n,x_{n-1},...,x_0) = f(x_n\mid x_{n-1},...,x_0)\cdot f(x_{n-1}\mid x_{n-2},...,x_0)\cdot f(x_{n-2}\mid x_{n-3},...,x_0) \cdot...\cdot f(x_0)$$

$$=\left(\prod_{i=1}^{n}\frac {1}{\sqrt{2\pi}}\exp\left\{-\frac {(x_i-\alpha x_{i-1})^2}{2}\right\}\right)\frac {1}{\sqrt{2\pi}}\exp\left\{-\frac {x_0^2}{2}\right\}$$

Viewed as a likelihood function of $\alpha$, and taking its natural logarithm, we have

$$\ln L(\alpha \mid \mathbf X) = -\frac 12\sum_{i=1}^n (x_i-\alpha x_{i-1})^2 +c$$

...where in $c$ is also included the density of $x_0$ (but $x_0$ affects estimation of $\alpha$ through its presence in the conditional density related to $X_1$).

Then

$$\frac {\partial \ln L(\alpha \mid \mathbf X)}{\partial \alpha} = \frac {\partial }{\partial \alpha} \left(-\frac 12\sum_{i=1}^n (x_i-\alpha x_{i-1})^2\right)$$

$$=-\frac 12\frac {\partial }{\partial \alpha} \left(\sum_{i=1}^n (x_i^2-2\alpha x_ix_{i-1}+\alpha^2x_{i-1}^2)\right)$$

$$=-\frac 12\frac {\partial }{\partial \alpha} \left(\sum_{i=1}^n x_i^2-2\alpha \sum_{i=1}^nx_ix_{i-1}+\alpha^2\sum_{i=1}^nx_{i-1}^2)\right)$$

$$=\sum_{i=1}^n x_ix_{i-1} -\alpha\sum_{i=1}^nx_{i-1}^2$$

Setting

$$\frac {\partial \ln L(\alpha \mid \mathbf X)}{\partial \alpha} =0\Rightarrow \hat \alpha_{ML} = \frac {\sum_{i=1}^n x_ix_{i-1}}{\sum_{i=1}^nx_{i-1}^2}$$

while $$\frac {\partial^2 \ln L(\alpha \mid \mathbf X)}{\partial \alpha^2} = -\sum_{i=1}^nx_{i-1}^2 <0$$

which guarantees a global and unique maximum, since it is negative irrespective of $\alpha$.

• Thanks a lot! Did not expect such a detailed explanation, but it's great! Commented Oct 14, 2013 at 11:51
• You're welcome Michael. I remember you are a biologist taking a stats course. How is it going so far? Commented Oct 14, 2013 at 13:15
• Hey Alecos! It's very interessting! But I am laking many mathematical tools. That is why I ask so many questions here. But the good thing is that I understand the stuff after solving the exercises ;-) Commented Oct 14, 2013 at 14:25