Application of Maximum Likelihood estimation (MLE) to the step of Feasible Generalized Least Square (FGLS)

I have the following regression

$$y = X\beta +u$$

where $$y$$ and $$u$$ are $$(n\times 1)$$ and $$X$$ is a fixed $$(n \times k)$$ matrix with full column rank and $$\beta$$ is an unknown $$(k\times 1)$$ vector of parameters.

$$E(u)=0$$ and $$var(u)=\sigma^2 V$$ with $$(n\times n)$$ symmetric and positive definite non-diagonal matrix $$V$$.

Let $$\hat {\beta_{OLS}}= (X'X)^{-1} X'y$$

Suppose $$u_t =\phi u_{t-1} +v_t$$ where $$v_t$$ is a white noise process with variance $$\sigma_v^2$$ and $$\phi$$ is an unknown parameter with $$|\phi|<1$$.

Describe how to get the feasible generalized least square (FGLS) estimator for $$\beta$$.

In the FGLS estimation procedure, we should use the maximized likelihood estimator $$\hat {\phi_{MLE}}$$, which should be obtained through the conditional log-likelihood function set conditional on the first observation.

What I did

I obtain the variance-covariance matrix as follows:

Since $$\phi$$ is unknown, the matrix $$V$$ is also unknown. Therefore, we need to use FGLS as follows:

Step 1: Estimate the generalized regression $$y=X\beta + u$$ by OLS and obtain $$\hat {u}$$.

Step 2: Construct the auxiliary regression $$\hat {u_t}= \phi \hat {u_{t-1}}+ v_t$$ in order to obtain $$\hat {\phi_{MLE}}$$ and $$\hat {v_{MLE}}$$.

For that, I write down a likelihood function

$$v_t \sim N(0, \sigma_v^2)$$

$$\hat {u_t} \sim N(0, \sigma_v^2)$$

$$I(\theta, u)=- \frac{n}{2} log(2\pi) - \frac{n}{2} log(\sigma_v^2) - \frac{n}{2\sigma_v^2} (\hat {u_t}- \phi \hat {u_{t-1}})'(\hat {u_t} - \phi \hat {u_{t-1}})$$

where $$\theta = [\phi \sigma_v^2]$$

FOC:

$$\frac{\partial I(\theta, u)}{\partial \phi} = 0$$

$$\hat {\phi_{MLE}} = (\hat {u_{t-1}}' \hat {u_{t-1}})\hat {u_{t-1}}' \hat {u_{t}}$$

Step 3:

By using $$\hat {\phi_{MLE}}$$ and $$\hat {v_{MLE}}$$, I will obtain the matrix $$V$$ and calculate $$\hat {\beta_{FGLS}} = (X'V^{-1}X)^{-1}X'V^{-1}y$$

I couldn’t do the step 2. The question states the conditional log-likelihood function, however I cannot write down this log-likelihood function as conditional function to solve for $$\hat {\phi_{MLE}}$$ and $$\hat {v_{MLE}}$$.

• Hi: Check out Andrew Harvey's "Econometric Analysis of Time Series". This book has beautiful, step by step explanations for how to construct MLE's for that type of problem. Commented Sep 18, 2022 at 5:23
• I have no chance to get this book. How can I find this book? Or can you share the related part of the book as an answer? I really need to learn how to solve for this question. Can you help me to solve this question? Thank you for your helps @mlofton
– 1190
Commented Sep 18, 2022 at 14:55
• I can send the relevant section to you through email but let me first see if I can find a pdf of the book on the internet. I bought it on amazon but Harvey's books are generally expensive. The title is "The Econometric Analysis of Time Series" by Andrew Harvey. Let me see if I can find it somewhere and get back to you. Commented Sep 19, 2022 at 15:33
• Hi: You can buy it for 3 dollars from Abe books. It says that condition is poor but who cares really. abebooks.com/Econometric-Analysis-Time-Series-Harvey-A/…-comus_shopp_textbook--naa-_-naa&msclkid=3d436d9c8eda1f4da91a0eb402d17799 Still let me see if I can find a pdf of the book on the net. Commented Sep 19, 2022 at 15:36
• This is not Harvey's but Pollock pretty much explains the same thing. le.ac.uk/users/dsgp1/COURSES/ELOMET/LECTURE5.PDF. Commented Sep 19, 2022 at 15:44

Note that $$\hat{u}_t = \phi \hat{u}_{t-1} + v_t,\, v_t \sim \mathcal N\left(0, \sigma^2_{v}\right)$$ implies $$\hat{u}_t | \hat{u}_{t-1}\sim \mathcal N\left(\phi \hat{u}_{t-1}, \sigma^2_{v}\right)$$ for $$t = 2, \ldots, n$$.
With the additional assumption $$\hat{u}_1 \sim \mathcal N\left(0, \sigma^2_{v}/\left(1 - \phi^2\right)\right)$$ you get the full log-likelihood kernel
$$l\left(\phi,\sigma^2_v;\hat{u}\right) = -\frac{n}{2} \cdot \ln\left(\sigma^2_v\right) + \frac{1}{2} \cdot \ln\left(1-\phi^2\right) - \frac{1}{2\sigma^2_v} \left[\sum_{t=2}^n\left(\hat{u}_t-\phi\hat{u}_{t-1}\right)^2 + \left(1 - \phi^2\right)\hat{u}_1^2 \right].$$ There is no closed-form solution for the maximum likelihood estimates $$\hat{\phi}$$ and $$\hat{\sigma}^2_v$$ in this case, but they can be determined iteratively.

Alternatively, you could treat $$\hat{u}_1$$ as deterministic/known and just maximize the conditional likelihood by maximizing the conditional log-likelihood kernel $$l_c\left(\phi,\sigma^2_v;\hat{u}\right) = -\frac{n-1}{2} \cdot \ln\left(\sigma^2_v\right) - \frac{1}{2\sigma^2_v} \sum_{t=2}^n\left(\hat{u}_t-\phi\hat{u}_{t-1}\right)^2.$$ The corresponding score equations \begin{align} \frac{\partial l_c}{\partial \phi}\left(\phi,\sigma^2_v;\hat{u}\right) = \frac{1}{\sigma^2_v} \sum_{t=2}^n\left(\hat{u}_t-\phi\hat{u}_{t-1}\right) \cdot \hat{u}_{t-1} \overset{!}{=} 0 \\ \iff \sum_{t=2}^n\left(\hat{u}_t-\phi\hat{u}_{t-1}\right) \cdot \hat{u}_{t-1} = 0 \end{align}
and \begin{align} \frac{\partial l_c}{\partial \sigma^2_v}\left(\phi,\sigma^2_v;\hat{u}\right) = -\frac{n-1}{2\sigma^2_v} + \frac{1}{2\left(\sigma^2_v\right)^2} \sum_{t=2}^n\left(\hat{u}_t-\phi\hat{u}_{t-1}\right)^2 \overset{!}{=} 0 \\ \iff -\left(n - 1\right) + \frac{1}{\sigma^2_v} \sum_{t=2}^n\left(\hat{u}_t-\phi\hat{u}_{t-1}\right)^2 = 0 \end{align}
then yield the maximum conditional likelihood estimates \begin{align} &\hat{\phi} = \frac{\sum_{t=2}^n\hat{u}_t\hat{u}_{t-1}}{\sum_{t=2}^n\hat{u}_{t-1}^2}, \\ &\hat{\sigma}^2_v = \frac{1}{n-1} \sum_{t=2}^n\left(\hat{u}_t-\hat{\phi}\hat{u}_{t-1}\right)^2, \end{align} from which you only need $$\hat{\phi}$$ to calculate $$\hat{\beta}_\text{FGLS}$$ since the matrix $$V$$ does not depend on $$\sigma^2_v$$.

Instead of the two-step FGLS procedure you could also use a (full) maximum likelihood approach with the original data that, under the assumption of a multivariate normal $$u$$, computes $$\left(\hat{\beta}, \hat{\sigma}^2, \hat{\phi}\right)$$ as maximizer of the likelihood $$\mathcal{L}\left(\beta, \sigma^2, \phi; y, X\right) = \left(2\pi\sigma^2\right)^{-n/2}|V|^{-1/2} \exp\left(-\frac{1}{2\sigma^2}\left(y-X\beta\right)^\top V^{-1} \left(y-X\beta\right)\right),$$ where \begin{align} &V = \frac{1}{1-\phi^2} \begin{pmatrix} 1 & \phi & \phi^2 & \cdots & \phi^{n-1}\\ \phi & 1 & \phi & \cdots & \phi^{n-2}\\ \phi^2 & \phi & 1 & \cdots & \phi^{n-3}\\ \vdots & \vdots & \vdots & \ddots & \vdots\\ \phi^{n-1} & \phi^{n-2} & \phi^{n-3}& \cdots & 1 \end{pmatrix},|V|=\left(1-\phi^2\right)^{-1},\\ &V^{-1} = \begin{pmatrix} 1 & -\phi & 0 & \cdots & 0 & 0\\ -\phi & 1+\phi^2 & -\phi & \cdots & 0 & 0\\ 0 & -\phi & 1+\phi^2 & \cdots & 0 & 0\\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots\\ 0 & 0 & 0 & \cdots & 1+\phi^2 & -\phi\\ 0 & 0 & 0 & \cdots & -\phi & 1 \end{pmatrix}. \end{align} Usually, this would be done by iteratively computing the maximizer of the log-likelihood kernel $$l\left(\beta, \sigma^2, \phi; y, X\right) = -\frac{n}{2} \cdot \ln\left(\sigma^2\right) + \frac{1}{2} \cdot \ln\left(1-\phi^2\right) - \frac{1}{2\sigma^2}\left(y-X\beta\right)^\top V^{-1} \left(y-X\beta\right).$$

• Thank you very much!
– 1190
Commented Sep 21, 2022 at 22:40
• @1190 You're welcome, I'm glad I could help. Commented Sep 22, 2022 at 11:52
• @statmerkur: very nice answer but, in your second approach, where do the $\hat{u_t}$ come from ? also, in your third approach, would you be using some non-linear optimization method like BFGS or something like that ? Thanks. Oh, basically, what I'm saying is that that I don't think there is a closed form MLE for the OP's problem ? Thanks. Commented Sep 22, 2022 at 18:17
• @mlofton The $\hat{u}_t$'s are the OLS residuals resulting from regressing $y$ on $X$. Yes, there is no closed-form MLE in the third approach$-$that's why I wrote "iteratively". I would go for a Newton-Raphson or quasi-Newton (e.g. BFGS) type of optimization algorithm. Commented Sep 22, 2022 at 18:49
• @statmerkur In the closed-form MLE in the third approach, only a 1D search over $\phi$ needs to be performed, as I describe in my answer, because to each specific value of $\phi$ corresponds a specific matrix $V_\phi$ and the intermediate solution for $\beta_{\phi}$ has an optimal closed-form GLS solution, i.e., for each specific value of $\phi$ we are maximizing the likelihood. Commented Sep 22, 2022 at 21:42

It would appear that Feasible Generalized Least Square (FGLS) should be used when the covariance matrix of the errors $$cov(u) = \sigma^2V$$ has a completely unknown form. However, in this problem, you have very specific prior knowledge of its form: it's the autocorrelation matrix of a 1st order autoregressive process, AR(1), which is a Toeplitz matrix. You even wrote this matrix explicitly.

Therefore, I feel the most appropriate and exact solution is to "directly" maximize the conditional log likelihood (conditioned on $$\phi$$), equivalently minimize the negative log likelihood conditioned on $$\phi$$, and then maximize over $$\phi$$:

For each specific value of $$\phi$$, the log likelihood of $$u$$ conditioned on $$\phi$$ is a multivariate Gaussian of $$u$$ ($$u_{\phi} = y - X\beta_{\phi}$$), but it's necessary to keep the $$det(V_{\phi})$$ term, as explained below. For a specific value of $$\phi$$, i.e., conditioned on $$\phi$$, this is maximized by the Generalized Least Squares (GLS): $$\beta_{\phi} = (X'V^{-1}_{\phi}X)^{-1}X'V^{-1}_{\phi}y$$.

But in order to maximize the overall unconditional (log) likelihood, it is necessary to compare likelihood values for different values of $$\phi$$, which requires to keep the $$det(V_{\phi})$$ term in the expression of the log likelihood.

Since the value of the unknown $$\phi$$ is in the bounded 1D range $$-1 < \phi < 1$$, it is a simple matter to maximize the unconditional likelihood over this range by a numerical 1D search (for example by a Fibonacci search for the maximum, etc.), or analytically by computing the $$det(V_{\phi})$$ as a function of $$\phi$$ and differentiating by $$\phi$$. (The analytical expression for $$det(V_{\phi})$$ as a function of $$\phi$$ can be obtained "easily", since $$V$$ has a very specific "easy" Toeplitz form.)

• Thank you a lot.
– 1190
Commented Sep 21, 2022 at 22:41
• You have to incorporate $\sigma^2$ (without which the (log-)likelihood can't be evaluated) as well. Maximizing the profile (log-)likelihood of $\phi$ (treating both $\beta$ and $\sigma^2$, whose MLEs have closed-form expressions for a fixed $\phi$, as nuisance parameters) in a one dimensional optimization would work. Commented Sep 24, 2022 at 11:26