0
$\begingroup$

I am using a mathematical function to estimate the covariance matrix for some process from the variances and then using this covariance matrix in a generalised least squares estimation of the slope and intercept of the data. (Although the covariance matrix comes from a function, it is not modified as a part of the GLS estimation.)

However, sometimes I get nonsense values (i.e. 10e123, when the slope should be ~1) for the slope. Adding a small amount of Gaussian noise to the variances when this happens seems to remove the problem, but it is a bit inelegant of a solution. I have checked and there is no noticeable difference between the covariance matrix condition number/determinant/condition number when I get the erroneous result and not.

I was wondering if anyone knows how I either (1) detect when I will get an erroneous result so that I can add the noise before the slope is calculated or (2) have any ideas how to remove the erroneous result in the first place (to this aim, I have tried smoothing the matrix and scaling the diagonal amount other things with no luck).

Improve this question
$\endgroup$
7
  • $\begingroup$ Either you may not know or simply didn't mention: GLS is an iterative estimation procedure, an EM algorithm, that alternately estimates the covariance and the regression parameters until stability. You can't trust gradient estimates on boundaries, or divergent solutions. Without any further details, you need to change your initial conditions or verify that a solution actually exists. $\endgroup$
    – AdamO
    Commented Mar 2, 2022 at 19:31
  • $\begingroup$ Secondly, you seem to use the word "gradient" in the sense of the slope parameter in a regression model, relating an input variable x with a response y. When you are essentially trying to perform a maximum likelihood routine, this gets dicey. The objective function that one is trying to solve is the zero of the derivative of the (log) likelihood. Newton-Raphson or iteratively reweighted least squares essentially uses the gradient of that function is employed to do that. I would suggest that "slope" is clearer when talking about the coefficient to $x$ in the linear model. $\endgroup$
    – AdamO
    Commented Mar 2, 2022 at 19:35
  • $\begingroup$ Sorry for the confusing terminology, I will clarify in the question. For the GLS comment, I was referring to generalised least squares (en.wikipedia.org/wiki/Generalized_least_squares as defined in the equation above "Properties") which I didn't think was iterative (though it is possible I am misunderstanding) $\endgroup$
    – arm61
    Commented Mar 2, 2022 at 19:38
  • $\begingroup$ No problem. Yes the process is iterative, read the Wiki especially the "feasible GLS" section - which is the default method in software. (although I could name offhand a dozen different ways to estimate the broadly defined GLS coefficient and covariance parameters.) And returning to the prior point: by virtue of being an iterative routine, it's prone to divergent and boundary solutions that can sometimes be remedied by assumption checking. $\endgroup$
    – AdamO
    Commented Mar 2, 2022 at 20:36
  • $\begingroup$ I am not preforming feasible GLS. I calculate the covariance matrix from my function once and then don’t change it unless I get a dodgy slope then I add some noise. I am calculating explicitly the equation I mention above. (I recognise that my language in the may have been a bit confusing as I should indicate that the covariance matrix is not modified iteratively). $\endgroup$
    – arm61
    Commented Mar 2, 2022 at 20:47

1 Answer 1

Reset to default
0
$\begingroup$

After some investigation, the variances were noisy resulting in a noisy covariance matrix and the observed instabilities. This has been resolved by smoothing the noisy variances to a function.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.