# Proving that overspecifying a linear model increases variance

Suppose that a random variable Y is defined as $Y=\beta_0 + \beta_1x + \epsilon$, where $var(\epsilon)=\sigma^2$, the error has zero mean, and is independent across observations. Suppose that we fit $\hat{Y}=\hat{\beta_0}+\hat{\beta_1}x+\hat{\beta_2}x^2+\hat{\beta_3}x^3$. I would like to show that $var(\hat{\beta_0}+\hat{\beta_1}x) \geq var(\tilde{\beta_0}+\tilde{\beta_1}x)$, where $(\tilde{\beta_0},\tilde{\beta_1})$ are the estimates obtained by fitting the model correctly. One of my homework assignments asked me to show this for a specific dataset, but I think it should be true generally.

It's easy to show that $var(\hat{\beta_1}) \geq var(\tilde{\beta_1})$ and so on, as this question seems to ask. But I'm finding it difficult to extend this result to $var(\hat{\beta_0}+\hat{\beta_1}x) \geq var(\tilde{\beta_0}+\tilde{\beta_1}x)$. We can break the terms apart into $var(\hat{\beta_0}+\hat{\beta_1}x)=var(\hat{\beta_0})+x^2var(\hat{\beta_1})+2cov(\hat{\beta_0},\hat{\beta_1}x)$, but how can I simplify the covariance term?

I attempted to use the law of total covariance. Covariance is bilinear, and the estimated $\beta_0$'s are equivalent, so $cov(\hat{\beta_0},\hat{\beta_1}x) \geq cov(\tilde{\beta_0},\tilde{\beta_1}x)$ is equivalent to $cov(\hat{\beta_0},(\hat{\beta_1}-\tilde{\beta_1})) \geq 0$. By the law of total covariance, \begin{align*} Cov(\hat{\beta_0},(\hat{\beta_1}-\tilde{\beta_1})x) &= E(cov(\hat{\beta_0},(\hat{\beta_1}-\tilde{\beta_1})x|Y)) + cov(E(\hat{\beta_0}|Y),E(\hat{\beta_1}-\tilde{\beta_1}x|Y)) \\ &= E(cov(\bar{Y},(\hat{\beta_1}-\tilde{\beta_1})x|Y)) + cov(E(\bar{Y}|Y),E((\hat{\beta_1}-\tilde{\beta_1})x|Y)) \\ \end{align*} $\bar{Y}|Y$ is a constant, so its covariance with any other variable is zero. The first term equals zero. The second term, though, doesn't seem like it has to be positive or negative in general. Does anyone know if there's a way to complete the proof? My approaches have focused on reducing the covariance term to a separable function of the individual beta variances. Maybe there's another method?

Not sure if you are familiar with matrices, but I am writing the general results. Suppose the true model is $Y=X_1\beta_1 +\epsilon$, but you fit the misspecified model $Y=X_1\beta_1 +X_2\beta_2+\epsilon =X\beta +\epsilon$, where $Y$ is $n \times 1$ vector, $X_1$ is $n \times p_1$ matrix, $X_2$ is $n \times p_2$, and $X=(X_1,X_2), \beta=(\beta_1^T,\beta_2^T)^T$, which are $n \times p, p \times 1$, respectively. The variance of $\hat{\beta_1}$ under the true model is $\sigma^2 (X_1^TX_1)^{-1}$, thus the variance of prediction $$var(X_1\hat{\beta_1}) =\sigma^2 X_1(X_1^TX_1)^{-1}X_1^T =\sigma^2 P_1,$$ $P_1$ is the hat matrix. Under the misspecified model, the variance of $\tilde{\beta_1}$ is the $p_1 \times p_1$ submatrix of $\sigma^2 (X^TX)^{-1}$, which can be shown to be $$(X_1^TX_1-X_1^TP_2X_1)^{-1}$$ using the results from matrix inverse of block form , where $P_2$ is the hat matrix corresponds to $X_2$. Thus ,$$var(X_1\tilde{\beta_1})\ = \sigma^2 (X_1(X_1^TX_1-X_1^TP_2X_1)^{-1}X_1^T) \ge \sigma^2 P_1$$.
• There seems to have some mistakes in your proof. e.g., are your thinking $\hat{\beta_0}=\tilde{\beta_0}=\bar{Y}$ ? – Statisfun Dec 10 '17 at 6:35
• I did assume $\hat{\beta_0}=\tilde{\beta_0}=\bar{Y}$. That's not true generally? – Anshu says Reinstate Monica Dec 10 '17 at 18:38
• Also, thank you for the answer! How did you go about showing that $X_{1}(X^{T}_{1}X_{1}−X^{T}_{1}P_{2}X_{1})^{−1}X^{T}_{1} \geq P_1$? – Anshu says Reinstate Monica Dec 10 '17 at 19:27
• A further question: I've managed to show that when we subtract the inequality's right hand side from the left hand side, the result is positive definite. But if the specific question asks whether $var(\tilde{\beta_0}+\tilde{\beta_1}x) \geq var(\hat{\beta_0}+\hat{\beta_1}x)$, wouldn't we need a more specific result than positive definiteness? Namely, that the diagonal elements of $var(X_{1}\tilde{\beta_1})-var(X_{1}\hat{\beta_1})$ are all nonnegative? – Anshu says Reinstate Monica Dec 10 '17 at 21:48