Proving that overspecifying a linear model increases variance - Cross Validated most recent 30 from stats.stackexchange.com 2019-07-21T15:27:05Z https://stats.stackexchange.com/feeds/question/317966 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/317966 1 Proving that overspecifying a linear model increases variance Anshu https://stats.stackexchange.com/users/142015 2017-12-09T18:57:53Z 2017-12-10T18:42:58Z <p>Suppose that a random variable Y is defined as $Y=\beta_0 + \beta_1x + \epsilon$, where $var(\epsilon)=\sigma^2$, the error has zero mean, and is independent across observations. Suppose that we fit $\hat{Y}=\hat{\beta_0}+\hat{\beta_1}x+\hat{\beta_2}x^2+\hat{\beta_3}x^3$. I would like to show that $var(\hat{\beta_0}+\hat{\beta_1}x) \geq var(\tilde{\beta_0}+\tilde{\beta_1}x)$, where $(\tilde{\beta_0},\tilde{\beta_1})$ are the estimates obtained by fitting the model correctly. One of my homework assignments asked me to show this for a specific dataset, but I think it should be true generally.</p> <p>It's easy to show that $var(\hat{\beta_1}) \geq var(\tilde{\beta_1})$ and so on, as <a href="https://stats.stackexchange.com/questions/248219/extra-variable-in-regression-model-increases-variance-of-parameter">this question</a> seems to ask. But I'm finding it difficult to extend this result to $var(\hat{\beta_0}+\hat{\beta_1}x) \geq var(\tilde{\beta_0}+\tilde{\beta_1}x)$. We can break the terms apart into $var(\hat{\beta_0}+\hat{\beta_1}x)=var(\hat{\beta_0})+x^2var(\hat{\beta_1})+2cov(\hat{\beta_0},\hat{\beta_1}x)$, but how can I simplify the covariance term?</p> <p>I attempted to use the law of total covariance. Covariance is bilinear, and the estimated $\beta_0$'s are equivalent, so $cov(\hat{\beta_0},\hat{\beta_1}x) \geq cov(\tilde{\beta_0},\tilde{\beta_1}x)$ is equivalent to $cov(\hat{\beta_0},(\hat{\beta_1}-\tilde{\beta_1})) \geq 0$. By the law of total covariance, \begin{align*} Cov(\hat{\beta_0},(\hat{\beta_1}-\tilde{\beta_1})x) &amp;= E(cov(\hat{\beta_0},(\hat{\beta_1}-\tilde{\beta_1})x|Y)) + cov(E(\hat{\beta_0}|Y),E(\hat{\beta_1}-\tilde{\beta_1}x|Y)) \\ &amp;= E(cov(\bar{Y},(\hat{\beta_1}-\tilde{\beta_1})x|Y)) + cov(E(\bar{Y}|Y),E((\hat{\beta_1}-\tilde{\beta_1})x|Y)) \\ \end{align*} $\bar{Y}|Y$ is a constant, so its covariance with any other variable is zero. The first term equals zero. The second term, though, doesn't seem like it has to be positive or negative in general. Does anyone know if there's a way to complete the proof? My approaches have focused on reducing the covariance term to a separable function of the individual beta variances. Maybe there's another method?</p> https://stats.stackexchange.com/questions/317966/-/318008#318008 2 Answer by Statisfun for Proving that overspecifying a linear model increases variance Statisfun https://stats.stackexchange.com/users/173339 2017-12-10T05:38:35Z 2017-12-10T05:38:35Z <p>Not sure if you are familiar with matrices, but I am writing the general results. Suppose the true model is $Y=X_1\beta_1 +\epsilon$, but you fit the misspecified model $Y=X_1\beta_1 +X_2\beta_2+\epsilon =X\beta +\epsilon$, where $Y$ is $n \times 1$ vector, $X_1$ is $n \times p_1$ matrix, $X_2$ is $n \times p_2$, and $X=(X_1,X_2), \beta=(\beta_1^T,\beta_2^T)^T$, which are $n \times p, p \times 1$, respectively. The variance of $\hat{\beta_1}$ under the true model is $\sigma^2 (X_1^TX_1)^{-1}$, thus the variance of prediction $$var(X_1\hat{\beta_1}) =\sigma^2 X_1(X_1^TX_1)^{-1}X_1^T =\sigma^2 P_1,$$ $P_1$ is the hat matrix. Under the misspecified model, the variance of $\tilde{\beta_1}$ is the $p_1 \times p_1$ submatrix of $\sigma^2 (X^TX)^{-1}$, which can be shown to be $$(X_1^TX_1-X_1^TP_2X_1)^{-1}$$ using the results from <a href="http://www.cs.nthu.edu.tw/~jang/book/addenda/matinv/matinv/" rel="nofollow noreferrer">matrix inverse of block form </a>, where $P_2$ is the hat matrix corresponds to $X_2$. Thus ,$$var(X_1\tilde{\beta_1})\ = \sigma^2 (X_1(X_1^TX_1-X_1^TP_2X_1)^{-1}X_1^T) \ge \sigma^2 P_1$$.</p>