# Proof that estimator in overfitted model is still unbiased

Assume that the true population model is given by $$y=x'\beta+\epsilon$$, where $$x$$ and $$\beta$$ are k-dimensional vectors, and suppose that when performing a linear regression, we accidentally overfit the model as: $$y=X\beta+Z\gamma+\epsilon,$$ where $$y$$ and $$\epsilon$$ are now n-dimensional vectors, (i.e. sample size is $$n$$), $$X$$ and $$Z$$ are regressor matrices where $$Z$$ is redundant, and $$\beta$$ and $$\gamma$$ are conformable parameter vectors.

I am attempting to show that the overfitted estimator for $$\beta$$, called $$b_{ofit}$$, is still unbiased for $$\beta$$, but I don't know how to interpret the value for $$y$$. I see two possibilities:

The first: $$E[b_{ofit}]=E[(X'X)^{-1}X'y]=E[(X'X)^{-1}X'(X\beta+Z\gamma+\epsilon)]$$

The second: $$E[b_{ofit}]=E[(X'X)^{-1}X'y]=E[(X'X)^{-1}X'(X\beta+\epsilon)]$$

I interpret the first as what we imagine to be the expected value of $$b_{ofit}$$, given our overfitted model, but the second is the true expected value of $$b_{ofit}$$.

Can someone explain what's happening here? Should I assume that the two are equal, i.e. that $$\gamma$$ has zero expectation?

• I gave a proof below that skips some steps in the algebra. If you want me to add more steps, let me know.
– dlnB
Commented Mar 23, 2019 at 23:09
• Orthogonality of $X$ and $Z$ is only important if we estimate $y=X \beta +\epsilon$ when the true model is $y=X \beta+Z \gamma +\epsilon$. If $X$ and $Z$ are not orthogonal, then we have omitted variable bias. He is asking about the reverse case, i.e. when we estimate $y=X \beta+Z \gamma +\epsilon$, but the true model is $y=X \beta +\epsilon$, in which case $\hat{\beta}$ is still unbiased, but is inefficient.
– dlnB
Commented Mar 24, 2019 at 0:15

The two expressions you write both follow the wrong logic for the problem you describe. The first tells us what happens if we estimate $$y_i=X_i \beta +\epsilon_i$$ when the true model is $$y_i=X_i\beta + Z_i \gamma + \epsilon_i$$, as you replaced $$y$$ with the latter, but your regressor matrix is still just $$X$$. The second approach is estimating the true model correctly.

Correct approach: Notice that our regressor matrix is $$[X Z],$$ while the true value of $$y$$ is given by $$y_i=X_i \beta +\epsilon_i$$. Therefore: $$\left[ \begin{array}{cc} \hat{\beta}_{overfit}\\ \hat{\gamma} \end{array} \right]=\left[ \begin{array}{cc} X'X& X'Z \\ Z'X & Z'Z \end{array} \right] ^{-1} \left[ \begin{array}{cc} X'\\ Z'\end{array} \right](X\beta+\epsilon) = \left[ \begin{array}{cc} X'X& X'Z \\ Z'X & Z'Z \end{array} \right] ^{-1} \left[ \begin{array}{cc} X'(X\beta+\epsilon)\\ Z'(X\beta+\epsilon) \end{array} \right].$$

If we write $$\left[ \begin{array}{cc} X'X& X'Z \\ Z'X & Z'Z \end{array} \right] ^{-1}=\left[ \begin{array}{cc} A & B \\ C & D \end{array} \right],$$ it follows $$\hat{\beta}_{overfit}=AX'(X\beta+\epsilon)+BZ'(X\beta+\epsilon).$$

Taking expectations, and using the fact that $$E(\epsilon|X,Z)=0$$, we get $$E(\hat{\beta}_{overfit})=AX'X\beta+BZ'X\beta.$$ Using the rules of block matrix inversion, we get $$A=(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}$$ $$B=-(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}X'Z(Z'Z)^{-1}Z'.$$ It follows $$E(\hat{\beta}_{overfit})=(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}X'X\beta -(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}X'Z(Z'Z)^{-1}Z'X\beta.$$ Combining terms, we get $$E(\hat{\beta}_{overfit})=(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}(X'X-X'Z(Z'Z)^{-1}Z'X)\beta.$$ Using the fact that $$(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}(X'X-X'Z(Z'Z)^{-1}Z'X)$$ is simply an identity matrix, we get $$E(\hat{\beta}_{overfit})=\beta.$$

• Do you use the Partitioned Regression Theorem to get the first expression for the estimators? Commented Mar 24, 2019 at 11:15
• Nvm it's just from the normal equation. Commented Mar 24, 2019 at 12:38
• Yeah the starting equation for $\left[ \begin{array}{cc} \hat{\beta}_{overfit}\\ \hat{\gamma} \end{array} \right]$ comes from the OLS formula with the usual $X$ replaced with $[X Z]$ and $y$ replaced with $(X \beta + \epsilon)$. If you have any other questions I'm happy to answer them.
– dlnB
Commented Mar 24, 2019 at 14:28
• Intuitively you could say that the overfitted model is like fitting a "true" model with $\gamma = 0$. Commented Apr 4, 2019 at 9:27