1
$\begingroup$

I have read in a book the next property:

$$E(x'_{over} \widehat{\beta}_{over)}=x'_{true}\beta$$

$$Var(x'_{over} \widehat{\beta}_{over})\geq Var(x'_{true} \widehat{\beta}_{true})$$

I do not know how to begin with the demonstration. Any help will be very useful.

$\endgroup$
1
2
$\begingroup$

This is easier to do using matrix notation. Your first, true model (model 1) is in matrix notation $$ Y = X \beta + E $$ Where we assume the error terms in $E$ is iid dstributed with zero mean and constant variance $\sigma^2$. The overparametrized model (model 2) is in matrix form $$ \DeclareMathOperator{\V}{\mathbb{V}} Y = X \beta + Z \gamma + E $$ with the same specification of $E$ and all the components og $\gamma$ has the true value of zero.

The estimator of $\beta$ in model 1 is $\hat{\beta} = (X^T X)^{-1}X^T Y$ with variance matrix $\V \hat{\beta} = \sigma^2 (X^T X)^{-1}$.

In model 2 the parameter estimator becomes $$ \widehat{\begin{pmatrix}\beta \\ \gamma \end{pmatrix}} = ([X \colon Z]^T [X \colon Z])^{-1} [X \colon Z]^T Y = \\ \begin{pmatrix} X^T X & X^T Z \\ Z^T X & Z^T Z \end{pmatrix}^{-1} \begin{pmatrix} X^T Y \\ Z^T Y \end{pmatrix} $$ Now, we are only interested in the first component of this, let us write that $\hat{\beta}_2$ ("$\beta$ estimated from model 2). Using formulas for the inverse of a block matrix (see https://en.wikipedia.org/wiki/Block_matrix) we can write the variance matrix of these estimator as $$ \tag{*} \V \hat{\beta}_2 = \sigma^2 (X^T X)^{-1} + \sigma^2 (X^T X)^{-1} X^T Z (Z^T Z - Z^T H Z)^{-1} Z^T X (X^T X)^{-1} $$ where $H=X^T (X^T X)^{-1} X$ is the hat matrix for model 1. Now, we can see that the matrix in the second term here is positive definite (or, if the matrix $Z$ is not of full rank, positive semidefinite) so that the diagonal terms are positive, showing that the variance of each component of $\hat{\beta}_2$ is larger than the variance of the corresponding component of $\hat{\beta}$ (estimator from model 1).

EDIT

Here some more details as asked for. The block inverse formula we use is $$ \begin{pmatrix} A & B \\ C & D \end{pmatrix}^{-1} = \begin{pmatrix} A^{-1}+A^{-1} B (D -CA^{-1}B)^{-1} C A^{-1} & \cdot \\ \cdot & \cdot \end{pmatrix} $$ (where I have only given the part of the formula that we actually use). Now in that formula you use $$ A=X^T X, B= X^T Z, C= Z^T X, D= Z^T Z $$ which will give the results.

EDIT

To respond to the second question in comments: How can we conclude that the second term in (*) is positive definite? Note thet it has the general form $A B A^T$ where $B$ is positive definite, that is enough! Since $B$ is positive definite we have the cholesky decomposistion $B= C C^T$, using that we have $A B A^T = A C C^T A^T = AC (AC)^T$, and using that we can verify positive definiteness, let $x \not= 0$ be any vector, we get $$ x^T A B A^T x = x^T AC (AC)^T x = ((AC)^T x)^T ((AC)^T x) > 0 $$ since $y^T y>0$ for any nonzero vector. To use this we must show that the "middle matrix" $(Z^T Z - Z^T H Z)^{-1}$ is positive definite. First, the inverse of a positive definite matrix is positive definite so we must show that $Z^T Z - Z^T H Z$ is positive definite. Now $ Z^T Z - Z^T H Z = Z^T (I-H) Z = Z^T (I_H) (I-H) Z = ((I-H)Z)^T (I-H)Z$ and now this can be shown to be positive definite by the same "trick" we used above. Here we have used that $H$ is an orthogonal proyector matrix, so $I-H$ also is, so $I-H = (I-H)^2$.

$\endgroup$
8
  • 1
    $\begingroup$ Thanks a lot, @kjetil b halvorsen , i appreciate it a lot. Could you detail a bit more how you obtain the variance formula, please? $\endgroup$
    – Mike13
    Jun 4 '16 at 15:31
  • $\begingroup$ I actually understand what you are explaining. However, it is a big jump for me, to obtain the variance once we know that: $$A=XTX$$. Thanks a lot! :) @kjetil b halvorsen $\endgroup$
    – Mike13
    Jun 4 '16 at 16:08
  • $\begingroup$ I have reached to understand that: $$Vβ^2=σ2(XTX)−1+σ2(XTX)−1XTZ(ZTZ−ZTHZ)−1ZTX(XTX)$$. But it do not reach to understand the conclusion. Could you explain a bit more, please? Thanks! $\endgroup$
    – Mike13
    Jun 4 '16 at 17:54
  • $\begingroup$ This is because the variance of a vector is the variabce-covariance matrix? @kjetil b halvorsen $\endgroup$
    – Mike13
    Jun 4 '16 at 18:11
  • $\begingroup$ How do we know that $$2(XTX)−1XTZ(ZTZ−ZTHZ)−1ZTX(XTX)−1 V⁡β^2=σ2(XTX)−1+σ2(XTX)−1XTZ(ZTZ−ZTHZ)−1ZTX(XTX)−1$$ is positive definite? $\endgroup$
    – Mike13
    Jun 4 '16 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.