6
$\begingroup$

Say we have a full regression model \begin{align*} \mathbf{y} &= \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}\\ &= \mathbf{X}_p \boldsymbol{\beta}_p + \mathbf{X}_r \boldsymbol{\beta}_r + \boldsymbol{\epsilon}\\ \end{align*} and a smaller model $$ \mathbf{y} = \mathbf{X}_p \boldsymbol{\beta}_p + \widetilde{\boldsymbol{\epsilon}}, $$ where both $\boldsymbol{\epsilon}$ and $\widetilde{\boldsymbol{\epsilon}}$ have the covariance matrix $\sigma^2 \mathbf{I}$. Note that, if the full model is true, $\widetilde{\boldsymbol{\epsilon}} = \mathbf{X}_r \boldsymbol{\beta}_r + \boldsymbol{\epsilon}$

The estimate of $\boldsymbol{\beta}_p$ from the full model is $$ \hat{\boldsymbol{\beta}}_p^* = \left[(\mathbf{X}^\intercal\mathbf{X})^{-1} \mathbf{X}^\intercal \mathbf{y}\right]_p $$ and the estimate from the smaller model is $$ \hat{\boldsymbol{\beta}}_p = (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{y}. $$

The bias for the full model is $\mathbf{0}$, and the bias for the small model is $$ (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r \boldsymbol{\beta}_r = \mathbf{A}\boldsymbol{\beta}_r . $$

The covariance matrix for the full model's estimates is $$ V[\hat{\boldsymbol{\beta}}^*_p] = \sigma^2 \left[(\mathbf{X}^\intercal\mathbf{X})^{-1}\right]_p $$ and the covariance matrix for the smaller model's estimates is $$ V[\hat{\boldsymbol{\beta}}_p] = \sigma^2 (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1}. $$ Note that the difference between these is positive semi-definite by the formula for inverses of block matrices.

What are all of the situations where $\text{MSE}[\hat{\boldsymbol{\beta}}^*_p] - \text{MSE}[\hat{\boldsymbol{\beta}}_p]$ is positive semi-definite?

  • One is when $\mathbf{X}_p^\intercal \mathbf{X}_r = 0$. Then both biases are zero, and $$ V[\hat{\boldsymbol{\beta}}^*_p] - V[\hat{\boldsymbol{\beta}}_p] $$ is positive semidefinite because, as Yves points out, the variances are the same because the estimates are the same.

Are there any others? This book mentions that it is also true when $$ V[\hat{\boldsymbol{\beta}}^*_r] - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal $$ is positive semi-definite, but I haven't been able to show this.

Improve this question
$\endgroup$
14
  • 1
    $\begingroup$ Missing link for the book? When $\mathbf{X}_p^\top \mathbf{X}_r= \mathbf{0}$ I believe that the two estimates are identical (the matrix $\mathbf{X}^\top\mathbf{X}$ is then block diagonal) so they have the same covariance. $\endgroup$
    – Yves
    Jun 5 '19 at 8:30
  • 2
    $\begingroup$ There is a critical ambiguity in this otherwise well-formulated question: the "$\epsilon$" in the full model is not the same as the "$\epsilon$" in the smaller model. Thus, towards the end, exactly what do the symbols "$\sigma$" refer to?? (Do they even represent the same quantity?) Your answer to that will clarify what assumptions you are making as well as clarify the question itself. $\endgroup$
    – whuber ♦
    Jun 5 '19 at 12:03
  • $\begingroup$ @Yves yes that’s true. I will post a link shortly; thanks. $\endgroup$
    – Taylor
    Jun 5 '19 at 17:05
  • $\begingroup$ @whuber you're correct. I just edited the post. $\endgroup$
    – Taylor
    Jun 5 '19 at 17:13
  • 1
    $\begingroup$ Consider an extreme case where the $\mathbf{X}_r$ suffice for a perfect fit, so that the variance of the $\tilde\epsilon$ is essentially zero. Indeed, the reduction in error variance is the key component of a standard measure of what is accomplished by including the extra variables (namely, the $F$ statistic). That ought to convince you of the merits of distinguishing the two variances! $\endgroup$
    – whuber ♦
    Jun 5 '19 at 17:20
1
$\begingroup$

The sufficient condition mentioned in the book turns out to be necessary as well! I finally verified it using the two different formulas for the inverse of a block matrix.

Looking at the full model estimator $$ \mathbf{X}^\intercal\mathbf{X} = \begin{bmatrix} \mathbf{X}_p^\intercal\mathbf{X}_p &\mathbf{X}_p^\intercal\mathbf{X}_r \\ \mathbf{X}_r^\intercal\mathbf{X}_p & \mathbf{X}_r^\intercal\mathbf{X}_r \end{bmatrix} $$ so $\text{MSE}[\hat{\boldsymbol{\beta}}^*_p]=\sigma^2[(\mathbf{X}^\intercal\mathbf{X})^{-1}]_p$ equals $$ \sigma^2\left\{(\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} + (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1}\mathbf{X}^\intercal_p\mathbf{X}_r(\mathbf{X}_r^\intercal\mathbf{X}_r - \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r)^{-1}\mathbf{X}_r^\intercal\mathbf{X}_p(\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1}\right\}. \tag{1} $$

Looking at the smaller model's estimator, $\mathbf{A} = (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1}\mathbf{X}_p^\intercal \mathbf{X}_r$ so \begin{align*} \text{MSE}[\hat{\boldsymbol{\beta}}_p] &= V[\hat{\boldsymbol{\beta}}_p] + \mathbf{A}\boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal \mathbf{A}^\intercal\\ &= \sigma^2(\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1} + (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1}\mathbf{X}_p^\intercal \mathbf{X}_r \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1} \tag{2}\\ \end{align*}

Subtracting (2) from (1) yields $$ (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1}\mathbf{X}_p^\intercal \mathbf{X}_r\left[ \sigma^2 (\mathbf{X}_r^\intercal\mathbf{X}_r - \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r)^{-1} - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal \right]\mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1} $$ which is positive semi-definite if this is as well: $$ \sigma^2 (\mathbf{X}_r^\intercal\mathbf{X}_r - \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r)^{-1} - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal. $$ This is the same as the expression $V[\hat{\boldsymbol{\beta}}^*_r] - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This answer seems to rely on the untenable assumption that the two error variances are the same. That looks like a fundamental mistake to me: although it works mathematically, it has no application at all. $\endgroup$
    – whuber ♦
    Jun 6 '19 at 15:12
  • $\begingroup$ @whuber I don’t see why. One is just a location-shifted version of the other. $\endgroup$
    – Taylor
    Jun 6 '19 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.