In a standard regression problem

\begin{equation} \mathbf{y} = \mathbf{X} \beta + \mathbf{e} \ , \end{equation}

the solution to $\beta$ when the system is overdetermined is $\hat{\beta}= \left(\mathbf{X}\Sigma^{-1}\mathbf{X}\right)^{-1}\mathbf{X}^T\Sigma^{-1}\mathbf{y}$, where $\Sigma = \sigma^{-2} \operatorname{Var}(\mathbf{e})$.

The solution to an underdetermined system when $\Sigma\equiv \mathbf{I}$ is $\hat{\beta} = \mathbf{X}^T\left(\mathbf{X}\mathbf{X}^T\right)^{-1}\mathbf{y}$. Can a similar formula to the least squares case be written when $\Sigma \ne \mathbf{I}$ such that $\hat{\beta} = \mathbf{X}^T\Sigma^{-1}\left(\mathbf{X}\mathbf{X}^T\Sigma^{-1}\right)^{-1} \mathbf{y}$?

I have not been able to find any good references on this system and would appreciate if you could suggest any also.

  • 1
    $\begingroup$ Could you please clarify what you are asking? I do not see any "formulations" in your question--only formulas--so the meaning of "similar formulation to the least squares case" is not apparent. $\endgroup$
    – whuber
    Commented Jan 20, 2014 at 16:30
  • $\begingroup$ My apologies. I meant formulation in the construction of a formula but did not mean anything more abstract. I have changed the the word to say "formula" rather than "formulation." $\endgroup$
    – hatmatrix
    Commented Jan 20, 2014 at 16:51
  • 2
    $\begingroup$ Note that your expression for $\hat{\beta}$ in the line starting with "The solution to..." is incorrect; you need to post-multiply by $y$. $\endgroup$
    – jbowman
    Commented Jan 20, 2014 at 21:47

1 Answer 1


Yes, it can be derived in the same way as the Generalized Least Squares estimator.

To customize the result to an underdetermined system, note the derivation of GLS as the best linear unbiased estimator uses the factorization $\Sigma = AA^T$ as follows. Define $Y^* = A^{-1}Y$, $X^*=A^{-1}X$, and $e^*=A^{-1}e$. Then the OLS solution to $Y^* = X^*\beta + e^*$ satisfies the Gauss-Markov requirements, and is consequently the best linear unbiased estimator.

When using the right generalized inverse $C^+ = C^T(CC^T)^{-1}$ to form the estimate $\hat{\beta} = X^+Y$ in the underdetermined case, you can just use $X^*$ and $Y^*$ instead of $X$ and $Y$. Writing it out in terms of $A$ and $X$ gives:

$\hat{\beta} = (A^{-1}X)^T(A^{-1}XX^T(A^{-1})^T)^{-1}A^{-1}Y$

  • $\begingroup$ The econometrics lecture notes was very helpful in addition. Thanks. $\endgroup$
    – hatmatrix
    Commented Jan 20, 2014 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.