1
$\begingroup$

Assume linear model $$Y = X \beta + \epsilon \\ \epsilon \sim \mathbb{N}(0, \Omega), $$ where $\Omega$ is a known covariance matrix. The GLS estimator for $\beta$ is well-known: $$\widehat{\beta}_{GLS} = (X^T \Omega^{-1} X)^{-1}X^T \Omega^{-1} y$$ Now if we further assume $X$ to be a square matrix and full-rank, we can infer an updated formula for $\widehat{\beta}_{GLS}$:

$$\widehat{\beta}_{GLS} = X^{-1} \Omega X^{-T}X^T \Omega^{-1} y = X^{-1}y$$

And it's just an OLS estimator that does not in any way take the known covariance $\Omega$ into an account.

I naturally would expect $\widehat{\beta}_{GLS}$ to be corrected for $\Omega$ as it is "hardcoded" into the equation of a linear model, but instead, it is the same as $\widehat{\beta}_{OLS}$; the residuals produced by this estimator are zeros. I could argue that there was a parametric multiplier in front of the covariance matrix $\sigma^2 \Omega$ and then conclude that $\sigma = 0$, but, well, there is none. In a sense, I see how it might be ok, as residuals are exactly at the mean of $\epsilon$ and therefore the likelihood is high.

However, I don't think I really understand the intuition behind it and it looks "broken" to me. Is there anything that can be done about it?

Improve this question
$\endgroup$
3
  • $\begingroup$ Hi: If the number of predictors is equal to the number of observations, then it's like a system of equations in algebra so there's no error. You need more observations than predictors in order for there to be error. $\endgroup$
    – mlofton
    Aug 4, 2020 at 1:49
  • $\begingroup$ I understand this, what I fail to see is that despite me imposing an error term with strict covariance matrix it is still zero. The thing is that my intuition tells me that $\Omega$ should play a regularization-like role. $\endgroup$
    – Geo
    Aug 4, 2020 at 20:20
  • $\begingroup$ Hi: Now., I understand better what you're saying and it's an interesting issue. But I guess that I would think of it this way: There's only one solution so, the value of the covariance matrix is irrelevant in the problem. No matter how small or how large it's elements are , the solution is the same and exact. It can't act as a regularizer because there's only one solution. Hopefully others can say more but that's my take. $\endgroup$
    – mlofton
    Aug 5, 2020 at 13:47

1 Answer 1

Reset to default
0
$\begingroup$

Geo: I was thinking about this more and I think it goes like this. When the number of observations is greater than the number of predictors then you have room to move around when trying to project the response on to the basis space of the $X$ matrix. The covariance matrix $\Omega$ helps one to decide where to move because it provides information on how the response variable are related to each other or how variable they are individually. But, again, in the case where the predictors equals the number of observations, there is no room to move. In fact, there is nothing to project because the response is already in the basis space of the $X$ matrix. So, the covariance matrix, $\Omega$, is not helpful in this case.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I believe this is a good explanation. Just a note (not really related to your answer): the behaviour I initially expected can be modelled via imposing a bayes prior on $X \beta$, namely $\mathbb{N}(y, \Omega)$. $\endgroup$
    – Geo
    Aug 5, 2020 at 17:22
  • $\begingroup$ Thanks Geo. Are you saying that this is the case even if the number of predictors is equal to the number of observations ? I would think not because there's still only one solution. Of course, corrections to my lack of understanding are appreciated. $\endgroup$
    – mlofton
    Aug 5, 2020 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.