3
$\begingroup$

First of all, I have to admit that I am not statistician so some of my nomenclature could not be very rigorous and maybe a bit confusing; pleas ask me to clarify if necessary.

The Problem

Let's say I want to solve a linear system $Ax = b$ where my vector $b$ has some uncorrelated and non-constant Gaussian noise (i.e. noise variance depends on the entry of $b$). As far as I know, the optimal way of solving this system in terms of minimum least-squares error is to use weighted least squares. Therefore, $$x_w = (A^TW^{-1}A)^{-1}A^TW^{-1}b$$ gives the optimal solution where $W$ is a diagonal matrix with the variance of each entry of $b$.

Let's also say that I can perform a linear transform $B$ to the system which will correlate my noise, $$BAx = Bb,$$ therefore now I need to solve the system using generalized least squares, $$x_g = ((BA)^TG^{-1}BA)^{-1}(BA)^TG^{-1}b,$$ where $G = BGB^T$ is the covariance matrix.

My questions

In practice, I can solve the problem in these two ways, but I do not know which solution will have lower residual, so my questions are:

  • Is there any transform $B$ that will make $x_g$ have less 'error' than $x_w$?
  • Are there error estimations for weighted and generalized least squares solutions? Do they depend on the patterns of $W$ and $G$? For example, a generalized least squared solution with a full $G$ matrix will have higher or lower error than a weighted least square solution where the diagonal matrix $W$ has large values (variance) at the diagonal?

What I want to do is to see if linear transforming my problem (with the colateral effect of correlating the noise) could in some cases give me better estimators of the solution.

PD: I do not ask for a long solution but just some references and guidance in order to solve this concern that I have.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The only way to answer your questions is to run a series of numerical experiments. Simply: take some concrete matrix A and vector b, run the linear solver, evaluate the obtained least squares solution, see what parts of the solution are unacceptable from the numerical point of view, then begin experimenting with other matrices. Perhaps you could find something on the following webpage; it addresses a least squares approach to image deblurring: http://members.ozemail.com.au/~comecau/CMA_LS_Sparse.htm

If you would like to participate in a discussion, here is a blog with a page devoted to the least squares processing: http://comecau.blogspot.com/2018_09_20_archive.html

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Hi: They are really two different issues. the case where the variance is uncorrelated but non constant is estimated using the results of halbert white. I think it's econometrica, mid 1980's. title is something like "a heteroscedastic consistent estimate of covariance matrix". second case you describe is just regular GLS ( assuming you mean that noise is correlated but constant ) and that was derived by Aitken in the 60's I think. At one point it was called the Aitken estimator. $\endgroup$
    – mlofton
    Oct 6, 2018 at 10:18
  • 1
    $\begingroup$ Here's a bio on White. bateswhite.com/newsroom-news-38.html. Sadly, he died pretty young. Otherwise, who knows what else he would have done. He has a pretty decent book called "asymptotic theory for econometricians". Fairly rigorous but, if you want all the gory details, one really needs to grind through his papers. Not an easy task but probably the only way to truly follow it. $\endgroup$
    – mlofton
    Oct 6, 2018 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.