# Has anyone studied linear regression where the covariance matrix of the error is a function of the parameters being estimated?

Consider the multivariate linear model: $$y = X\beta + e$$ $$y$$ is the measured output, $$X$$ is the model matrix, $$\beta$$ is the parameter vector, and $$e$$ is the zero-mean error vector: $$E[e] = 0 \qquad E[ee^T] = V$$ The goal is to estimate $$\beta$$ from $$y$$. It is well known that the minimum variance, affine unbiased estimator of $$\beta$$ exits if and only if $$X$$ has linear independent columns, in which case it is $$\hat\beta = [X^T (V + XX^T)^+ X]^{-1} X^T (V + XX^T)^+y$$ Many statisticians, mathematicians, econometricians, etc. have studied this problem where the error covariance matrix $$V$$ is partially or completely unknown. There are plenty of approaches on how to estimate $$\beta$$ and/or $$V$$ in this case.

My situation: I have an application where $$V$$ is unknown, but $$V$$ is a known function of $$\beta$$: $$V = f(\beta)$$ I have conducted a literature search looking for sources that have studied this problem, but I haven't found anything. It seems this situation has not been widely considered, if at all, but I am not a statistician; the literature is vast and I could just be searching for the wrong thing. However, if those sources are out there, I would like to cite them.

My question: Does anyone know of any papers/books/whatever that consider this problem?

• Being a non-statistician myself I suggest that you look for the literature referring to this specific problem in the relevant field, rather than for a general solution by statisticians/mathematicians/etc. The problem might be well explored by the practitioners for the realistic choices of $f(\beta)$, even if a general solution is impossible. – Roger Vadim Nov 14 '19 at 17:02
• @Vadim I appreciate the comment. Your suggestion is exactly what I am trying to do; my issue is that I don't really know where to start looking for such sources, and that is where I need help. – teerav42 Nov 14 '19 at 21:18
• I suppose that there is an actual (not purely mathematical) problem in biology/economics/physics or another field that is behind the question? – Roger Vadim Nov 14 '19 at 21:25
• @Vadim actually, not really. The problem I am studying is noise covariance estimation for discrete time, time invariant linear systems. The work is not motivated by any particular physical system in particular. Essentially, $\beta$ is the covariance matrices of the noises which drive the system. This is the source of all of the uncertainty in the system, so this is why the case $V = f(\beta)$ occurs, in a nutshell. – teerav42 Nov 14 '19 at 21:53

## 1 Answer

This problem is actually quite ubiquitous. It occurs in regression models where there is heteroscedasticity that is described by some parametric function of the data. It also occurs in standard time-series models where there is an auto-regressive parameter on the error terms. There are certainly many books and articles that look at this type of problem, but you will probably need to narrow it down to the type of parametric dependence you are interested in. Unless you have a contrary interest, I would suggest you start by looking at Gaussian AR models, and then work your way up from there.

Without knowing more about your background (e.g., your existing mathematical and statistical knowledge) it is quite difficult to suggest a reference that is pitched at the right level. If you have quite a good maths background, you might be able to read a useful primer on the linear AR(1) model by Grunwald, Hyndman and Tedesco (1996). This article examines various formulations of the linear AR(1) model, and looks at some relevant literature.

• Thanks! As a non-statistician, that is a bit of a mouthful for me. Without going into too much detail, what I am doing does involve a Gaussian AR model (though I'd just call it a discrete time linear model, i.e. $x_{k+1} = Ax_k + w_k$ and $y_k = Cx_k + v_k$. Can you suggest a reference I can start with? I know how I want to tackle the estimation problem for my application, so I don't care a ton about the details. Rather, I was about to write "the problem where $V = f(\beta)$ seems not to have been considered by many authors", so if that's not actually true, I just want to cite an example. – teerav42 Nov 14 '19 at 6:37