1
$\begingroup$

In restricted linear model, we have $$Y = X\beta+\varepsilon$$ $$A^T\beta=b$$

Where $A^T\beta$ is estimable (i.e., there exists a matrix $D$, such that $D^TX=A^T$) and $A$ is a $p\times q$ matrix with rank($A$)$=q$.

The estimation of $\beta$ is

$$\hat\beta = \beta_0 - (X^TX)^-A(A^T(X^TX)^-A)^{-1}(A^T\beta_0-b),$$ where $\beta_0=(X^TX)^-X^TY$.

My question is why $A^T(X^TX)^-A$ has a inverse. I know $A^T(X^TX)^-A$ is equal to the covariance matrix $\text{cov}(A^T(X^TX)^-X^TY)$ here. So could we prove that the covariance matrix is positive definite? Could any one tell me how to do it please? Thanks.

$\endgroup$
11
  • 1
    $\begingroup$ Because covariance matrix of any vector is positive definite? $\endgroup$ Commented Jun 21, 2018 at 13:30
  • 1
    $\begingroup$ @Richard Hardy All covariance matrices are positive semidefinite, not necessarily positive definite. $\endgroup$ Commented Jun 21, 2018 at 13:33
  • $\begingroup$ Thanks. The covariance matrix can be semi-positive definite. I thin the condition rank$(A) = q$ can ensure the covariance matrix be positive definite, but I dont know why $\endgroup$
    – coolcat
    Commented Jun 21, 2018 at 13:35
  • $\begingroup$ If the claim is true, you need to show that the covariance is full rank, i.e., nonsingular, which would make it positive definite. $\endgroup$ Commented Jun 21, 2018 at 13:40
  • 1
    $\begingroup$ What are $A^T$ and $b$? Are they fixed or are they some variable? What is the meaning of $(X^TX)^-$ it is not an ordinary inverse is it? Otherwise you get to $A^T(X^TX)^{-1}X^TY=b$. Are you talking about minimizing $Y-X\hat\beta$ restricted to $A^T\hat\beta=b$? $\endgroup$ Commented Jun 21, 2018 at 13:52

1 Answer 1

1
$\begingroup$

The covariance matrix in a restricted least squares problem is not positive definite as it is not full rank; the coefficients have to satisfy the linear constraints, e.g., $\beta_1 + \beta_2 = 0$, and this is sufficient to ensure that the rank of the covariance matrix is $p-q$ where there are $p$ variables and $q$ linear constraints (unless your constraints are redundant, which we will assume they are not) while having dimension $p \times p$.

You have slightly misapplied the generalized inverse as well. $X^TX$ is full rank (by assumption) so the usual inverse $(X^TX)^{-1}$ exists.

The actual covariance matrix of $\hat{\beta}$ is not $A^T(X^TX)^{-1}A$ but:

$$\sigma^2(X^TX)^{-1}\left(I - A[A^T(X^TX)^{-1}A]^{-1}A^T(X^TX)^{-1}\right)$$

a derivation of which can be found in many places, including http://www2.econ.iastate.edu/classes/econ671/hallam/documents/Restricted_Testing_000.pdf.

$\endgroup$
4
  • $\begingroup$ Thanks. In this setting, we do not have $X^TX$ is full rank, so how to show $A^T(X^TX)^−A$ has a inverse? $\endgroup$
    – coolcat
    Commented Jun 21, 2018 at 18:24
  • $\begingroup$ Just use the Moore-Penrose generalized inverse en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse; it is the same as the regular inverse when the matrix is full rank, and possesses all the desirable properties when it isn't. You can't guarantee that $A^T(X^TX)^-A$ has full rank if the inner matrix does not; imagine that $A = I$ for example and you can see that the regular inverse won't exist if $X$ is less than full rank. $\endgroup$
    – jbowman
    Commented Jun 21, 2018 at 18:29
  • $\begingroup$ We assume that $A^T = D^T X$, so $A^T(X^TX)^−A = D^TX(X^TX)^−X^TD =\text{cov}(A^T(X^TX)−X^TY)$. Could it be used to solve the problem? $\endgroup$
    – coolcat
    Commented Jun 21, 2018 at 18:37
  • $\begingroup$ But without assumptions about $D$ you're still doomed. You can't, for example, ensure that $A \neq I$, which it can be by making the "first part" of $D^T$ equal to the inverse of the "first part" of $X$ and the rest of $D^T$ equal to $0$. $\endgroup$
    – jbowman
    Commented Jun 21, 2018 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.