# When does the underfitted regression model have more precise coefficient estimates?

Say we have a full regression model \begin{align*} \mathbf{y} &= \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}\\ &= \mathbf{X}_p \boldsymbol{\beta}_p + \mathbf{X}_r \boldsymbol{\beta}_r + \boldsymbol{\epsilon}\\ \end{align*} and a smaller model $$\mathbf{y} = \mathbf{X}_p \boldsymbol{\beta}_p + \widetilde{\boldsymbol{\epsilon}},$$ where both $$\boldsymbol{\epsilon}$$ and $$\widetilde{\boldsymbol{\epsilon}}$$ have the covariance matrix $$\sigma^2 \mathbf{I}$$. Note that, if the full model is true, $$\widetilde{\boldsymbol{\epsilon}} = \mathbf{X}_r \boldsymbol{\beta}_r + \boldsymbol{\epsilon}$$

The estimate of $$\boldsymbol{\beta}_p$$ from the full model is $$\hat{\boldsymbol{\beta}}_p^* = \left[(\mathbf{X}^\intercal\mathbf{X})^{-1} \mathbf{X}^\intercal \mathbf{y}\right]_p$$ and the estimate from the smaller model is $$\hat{\boldsymbol{\beta}}_p = (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{y}.$$

The bias for the full model is $$\mathbf{0}$$, and the bias for the small model is $$(\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r \boldsymbol{\beta}_r = \mathbf{A}\boldsymbol{\beta}_r .$$

The covariance matrix for the full model's estimates is $$V[\hat{\boldsymbol{\beta}}^*_p] = \sigma^2 \left[(\mathbf{X}^\intercal\mathbf{X})^{-1}\right]_p$$ and the covariance matrix for the smaller model's estimates is $$V[\hat{\boldsymbol{\beta}}_p] = \sigma^2 (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1}.$$ Note that the difference between these is positive semi-definite by the formula for inverses of block matrices.

What are all of the situations where $$\text{MSE}[\hat{\boldsymbol{\beta}}^*_p] - \text{MSE}[\hat{\boldsymbol{\beta}}_p]$$ is positive semi-definite?

• One is when $$\mathbf{X}_p^\intercal \mathbf{X}_r = 0$$. Then both biases are zero, and $$V[\hat{\boldsymbol{\beta}}^*_p] - V[\hat{\boldsymbol{\beta}}_p]$$ is positive semidefinite because, as Yves points out, the variances are the same because the estimates are the same.

Are there any others? This book mentions that it is also true when $$V[\hat{\boldsymbol{\beta}}^*_r] - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal$$ is positive semi-definite, but I haven't been able to show this.

• Missing link for the book? When $\mathbf{X}_p^\top \mathbf{X}_r= \mathbf{0}$ I believe that the two estimates are identical (the matrix $\mathbf{X}^\top\mathbf{X}$ is then block diagonal) so they have the same covariance. – Yves Jun 5 at 8:30
• There is a critical ambiguity in this otherwise well-formulated question: the "$\epsilon$" in the full model is not the same as the "$\epsilon$" in the smaller model. Thus, towards the end, exactly what do the symbols "$\sigma$" refer to?? (Do they even represent the same quantity?) Your answer to that will clarify what assumptions you are making as well as clarify the question itself. – whuber Jun 5 at 12:03
• @Yves yes that’s true. I will post a link shortly; thanks. – Taylor Jun 5 at 17:05
• @whuber you're correct. I just edited the post. – Taylor Jun 5 at 17:13
• Consider an extreme case where the $\mathbf{X}_r$ suffice for a perfect fit, so that the variance of the $\tilde\epsilon$ is essentially zero. Indeed, the reduction in error variance is the key component of a standard measure of what is accomplished by including the extra variables (namely, the $F$ statistic). That ought to convince you of the merits of distinguishing the two variances! – whuber Jun 5 at 17:20

Looking at the full model estimator $$\mathbf{X}^\intercal\mathbf{X} = \begin{bmatrix} \mathbf{X}_p^\intercal\mathbf{X}_p &\mathbf{X}_p^\intercal\mathbf{X}_r \\ \mathbf{X}_r^\intercal\mathbf{X}_p & \mathbf{X}_r^\intercal\mathbf{X}_r \end{bmatrix}$$ so $$\text{MSE}[\hat{\boldsymbol{\beta}}^*_p]=\sigma^2[(\mathbf{X}^\intercal\mathbf{X})^{-1}]_p$$ equals $$\sigma^2\left\{(\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} + (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1}\mathbf{X}^\intercal_p\mathbf{X}_r(\mathbf{X}_r^\intercal\mathbf{X}_r - \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r)^{-1}\mathbf{X}_r^\intercal\mathbf{X}_p(\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1}\right\}. \tag{1}$$
Looking at the smaller model's estimator, $$\mathbf{A} = (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1}\mathbf{X}_p^\intercal \mathbf{X}_r$$ so \begin{align*} \text{MSE}[\hat{\boldsymbol{\beta}}_p] &= V[\hat{\boldsymbol{\beta}}_p] + \mathbf{A}\boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal \mathbf{A}^\intercal\\ &= \sigma^2(\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1} + (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1}\mathbf{X}_p^\intercal \mathbf{X}_r \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1} \tag{2}\\ \end{align*}
Subtracting (2) from (1) yields $$(\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1}\mathbf{X}_p^\intercal \mathbf{X}_r\left[ \sigma^2 (\mathbf{X}_r^\intercal\mathbf{X}_r - \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r)^{-1} - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal \right]\mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal \mathbf{X}_p)^{-1}$$ which is positive semi-definite if this is as well: $$\sigma^2 (\mathbf{X}_r^\intercal\mathbf{X}_r - \mathbf{X}_r^\intercal \mathbf{X}_p (\mathbf{X}_p^\intercal\mathbf{X}_p)^{-1} \mathbf{X}_p^\intercal \mathbf{X}_r)^{-1} - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal.$$ This is the same as the expression $$V[\hat{\boldsymbol{\beta}}^*_r] - \boldsymbol{\beta}_r\boldsymbol{\beta}_r^\intercal$$.