# Problem with estimation

I have read in a book the next property:

$$E(x'_{over} \widehat{\beta}_{over)}=x'_{true}\beta$$

$$Var(x'_{over} \widehat{\beta}_{over})\geq Var(x'_{true} \widehat{\beta}_{true})$$

I do not know how to begin with the demonstration. Any help will be very useful.

• Jun 2 '16 at 11:15

This is easier to do using matrix notation. Your first, true model (model 1) is in matrix notation $$Y = X \beta + E$$ Where we assume the error terms in $E$ is iid dstributed with zero mean and constant variance $\sigma^2$. The overparametrized model (model 2) is in matrix form $$\DeclareMathOperator{\V}{\mathbb{V}} Y = X \beta + Z \gamma + E$$ with the same specification of $E$ and all the components og $\gamma$ has the true value of zero.

The estimator of $\beta$ in model 1 is $\hat{\beta} = (X^T X)^{-1}X^T Y$ with variance matrix $\V \hat{\beta} = \sigma^2 (X^T X)^{-1}$.

In model 2 the parameter estimator becomes $$\widehat{\begin{pmatrix}\beta \\ \gamma \end{pmatrix}} = ([X \colon Z]^T [X \colon Z])^{-1} [X \colon Z]^T Y = \\ \begin{pmatrix} X^T X & X^T Z \\ Z^T X & Z^T Z \end{pmatrix}^{-1} \begin{pmatrix} X^T Y \\ Z^T Y \end{pmatrix}$$ Now, we are only interested in the first component of this, let us write that $\hat{\beta}_2$ ("$\beta$ estimated from model 2). Using formulas for the inverse of a block matrix (see https://en.wikipedia.org/wiki/Block_matrix) we can write the variance matrix of these estimator as $$\tag{*} \V \hat{\beta}_2 = \sigma^2 (X^T X)^{-1} + \sigma^2 (X^T X)^{-1} X^T Z (Z^T Z - Z^T H Z)^{-1} Z^T X (X^T X)^{-1}$$ where $H=X^T (X^T X)^{-1} X$ is the hat matrix for model 1. Now, we can see that the matrix in the second term here is positive definite (or, if the matrix $Z$ is not of full rank, positive semidefinite) so that the diagonal terms are positive, showing that the variance of each component of $\hat{\beta}_2$ is larger than the variance of the corresponding component of $\hat{\beta}$ (estimator from model 1).

EDIT

Here some more details as asked for. The block inverse formula we use is $$\begin{pmatrix} A & B \\ C & D \end{pmatrix}^{-1} = \begin{pmatrix} A^{-1}+A^{-1} B (D -CA^{-1}B)^{-1} C A^{-1} & \cdot \\ \cdot & \cdot \end{pmatrix}$$ (where I have only given the part of the formula that we actually use). Now in that formula you use $$A=X^T X, B= X^T Z, C= Z^T X, D= Z^T Z$$ which will give the results.

EDIT

To respond to the second question in comments: How can we conclude that the second term in (*) is positive definite? Note thet it has the general form $A B A^T$ where $B$ is positive definite, that is enough! Since $B$ is positive definite we have the cholesky decomposistion $B= C C^T$, using that we have $A B A^T = A C C^T A^T = AC (AC)^T$, and using that we can verify positive definiteness, let $x \not= 0$ be any vector, we get $$x^T A B A^T x = x^T AC (AC)^T x = ((AC)^T x)^T ((AC)^T x) > 0$$ since $y^T y>0$ for any nonzero vector. To use this we must show that the "middle matrix" $(Z^T Z - Z^T H Z)^{-1}$ is positive definite. First, the inverse of a positive definite matrix is positive definite so we must show that $Z^T Z - Z^T H Z$ is positive definite. Now $Z^T Z - Z^T H Z = Z^T (I-H) Z = Z^T (I_H) (I-H) Z = ((I-H)Z)^T (I-H)Z$ and now this can be shown to be positive definite by the same "trick" we used above. Here we have used that $H$ is an orthogonal proyector matrix, so $I-H$ also is, so $I-H = (I-H)^2$.

• Thanks a lot, @kjetil b halvorsen , i appreciate it a lot. Could you detail a bit more how you obtain the variance formula, please? Jun 4 '16 at 15:31
• I actually understand what you are explaining. However, it is a big jump for me, to obtain the variance once we know that: $$A=XTX$$. Thanks a lot! :) @kjetil b halvorsen Jun 4 '16 at 16:08
• I have reached to understand that: $$Vβ^2=σ2(XTX)−1+σ2(XTX)−1XTZ(ZTZ−ZTHZ)−1ZTX(XTX)$$. But it do not reach to understand the conclusion. Could you explain a bit more, please? Thanks! Jun 4 '16 at 17:54
• This is because the variance of a vector is the variabce-covariance matrix? @kjetil b halvorsen Jun 4 '16 at 18:11
• How do we know that $$2(XTX)−1XTZ(ZTZ−ZTHZ)−1ZTX(XTX)−1 V⁡β^2=σ2(XTX)−1+σ2(XTX)−1XTZ(ZTZ−ZTHZ)−1ZTX(XTX)−1$$ is positive definite? Jun 4 '16 at 18:26