# Minimizing an objective function with input variables from a correlated error term

I've been reading into how to minimize objective functions and I am curious about the following, I have a model $$y=X \beta +\epsilon$$ where $$E[\epsilon|X]=0$$ and $$Var[\epsilon|X]=\Sigma$$ where $$\Sigma$$ is not diagonal and is a covariance matrix. $$\beta$$ can be estimated by least squares to get, $$\hat{\beta}=(X^T \Sigma^{-1}X)^{-1}X^T\Sigma^{-1}y$$.

So this comes from the minimization of $$(y-X\beta)^T \Sigma^{-1}(y-X\beta)$$. Now if I assume that $$\epsilon \sim AR(2)$$ then $$Var[\epsilon|X]=\Sigma=Cov(AR(2))$$ making it a covariance matrix with respect to AR(2). So my question is, is there any way to decompose $$\Sigma$$ in the objective function such that it can be written in the form of the AR(2) coefficients and its variance so that the objective function can be minimized with respect to variance, and AR(2) coefficients if $$y, \beta$$, and $$X$$ are known.

Thank you

• Are you only interested in a "decomposition" of $\Sigma$, if so, what exactly do you mean by "decomposition"? Or do you want to understand the complete procedure of this optimization problem? Or do you just want to know an implementation you can use? Commented Sep 30, 2022 at 5:21
• Hi Frank, I actually understand most of the procedures of this optimization, however, I am curious in knowing if anything can be done with $\Sigma$, either rewritten in a different way say through QR decomposition or if it is legitimate to rewrite it as $\sigma^2_w* \Sigma$ since originally $\Sigma$ is calculated using the auto-covariance function which relies on $\sigma^2_w, \phi_1, \phi_2$ where $\sigma^2_w$ is just the variance of white noise, and $\phi_1, \phi_2$ are the coefficients values of the AR(2) model. Commented Sep 30, 2022 at 16:29

You can do linear regression for $$y = X\beta + \epsilon$$ using a covariance matrix for $$\epsilon$$ that belongs to an AR(2) model, or to any ARMA(p, q) model for that matter.
You can e.g. use the gls() function from the nlme R-package. It provides for various types of correlations, among them ARMA correlation (look for corARMA here). There, you provide the coefficients of the ARMA model which are then used by gls() to create the appropriate covariance matrix and execute the optimization.
• THanks, Frank, I'll have to check the gls() out as I haven't had to use it yet. Commented Oct 3, 2022 at 13:10