# In restrict linear model, why Cov($A^T(X^TX)^-X^TY$) is positive definite?

In restricted linear model, we have $$Y = X\beta+\varepsilon$$ $$A^T\beta=b$$

Where $A^T\beta$ is estimable (i.e., there exists a matrix $D$, such that $D^TX=A^T$) and $A$ is a $p\times q$ matrix with rank($A$)$=q$.

The estimation of $\beta$ is

$$\hat\beta = \beta_0 - (X^TX)^-A(A^T(X^TX)^-A)^{-1}(A^T\beta_0-b),$$ where $\beta_0=(X^TX)^-X^TY$.

My question is why $A^T(X^TX)^-A$ has a inverse. I know $A^T(X^TX)^-A$ is equal to the covariance matrix $\text{cov}(A^T(X^TX)^-X^TY)$ here. So could we prove that the covariance matrix is positive definite? Could any one tell me how to do it please? Thanks.

• Because covariance matrix of any vector is positive definite? Commented Jun 21, 2018 at 13:30
• @Richard Hardy All covariance matrices are positive semidefinite, not necessarily positive definite. Commented Jun 21, 2018 at 13:33
• Thanks. The covariance matrix can be semi-positive definite. I thin the condition rank$(A) = q$ can ensure the covariance matrix be positive definite, but I dont know why Commented Jun 21, 2018 at 13:35
• If the claim is true, you need to show that the covariance is full rank, i.e., nonsingular, which would make it positive definite. Commented Jun 21, 2018 at 13:40
• What are $A^T$ and $b$? Are they fixed or are they some variable? What is the meaning of $(X^TX)^-$ it is not an ordinary inverse is it? Otherwise you get to $A^T(X^TX)^{-1}X^TY=b$. Are you talking about minimizing $Y-X\hat\beta$ restricted to $A^T\hat\beta=b$? Commented Jun 21, 2018 at 13:52

The covariance matrix in a restricted least squares problem is not positive definite as it is not full rank; the coefficients have to satisfy the linear constraints, e.g., $\beta_1 + \beta_2 = 0$, and this is sufficient to ensure that the rank of the covariance matrix is $p-q$ where there are $p$ variables and $q$ linear constraints (unless your constraints are redundant, which we will assume they are not) while having dimension $p \times p$.

You have slightly misapplied the generalized inverse as well. $X^TX$ is full rank (by assumption) so the usual inverse $(X^TX)^{-1}$ exists.

The actual covariance matrix of $\hat{\beta}$ is not $A^T(X^TX)^{-1}A$ but:

$$\sigma^2(X^TX)^{-1}\left(I - A[A^T(X^TX)^{-1}A]^{-1}A^T(X^TX)^{-1}\right)$$

a derivation of which can be found in many places, including http://www2.econ.iastate.edu/classes/econ671/hallam/documents/Restricted_Testing_000.pdf.

• Thanks. In this setting, we do not have $X^TX$ is full rank, so how to show $A^T(X^TX)^−A$ has a inverse? Commented Jun 21, 2018 at 18:24
• Just use the Moore-Penrose generalized inverse en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse; it is the same as the regular inverse when the matrix is full rank, and possesses all the desirable properties when it isn't. You can't guarantee that $A^T(X^TX)^-A$ has full rank if the inner matrix does not; imagine that $A = I$ for example and you can see that the regular inverse won't exist if $X$ is less than full rank. Commented Jun 21, 2018 at 18:29
• We assume that $A^T = D^T X$, so $A^T(X^TX)^−A = D^TX(X^TX)^−X^TD =\text{cov}(A^T(X^TX)−X^TY)$. Could it be used to solve the problem? Commented Jun 21, 2018 at 18:37
• But without assumptions about $D$ you're still doomed. You can't, for example, ensure that $A \neq I$, which it can be by making the "first part" of $D^T$ equal to the inverse of the "first part" of $X$ and the rest of $D^T$ equal to $0$. Commented Jun 21, 2018 at 18:43