Assume that the true population model is given by $y=x'\beta+\epsilon$, where $x$ and $\beta$ are k-dimensional vectors, and suppose that when performing a linear regression, we accidentally overfit the model as: $$y=X\beta+Z\gamma+\epsilon,$$ where $y$ and $\epsilon $ are now n-dimensional vectors, (i.e. sample size is $n$), $X$ and $Z$ are regressor matrices where $Z $ is redundant, and $\beta$ and $\gamma$ are conformable parameter vectors.

I am attempting to show that the overfitted estimator for $\beta$, called $b_{ofit}$, is still unbiased for $\beta$, but I don't know how to interpret the value for $y$. I see two possibilities:

The first: $$E[b_{ofit}]=E[(X'X)^{-1}X'y]=E[(X'X)^{-1}X'(X\beta+Z\gamma+\epsilon)] $$

The second: $$E[b_{ofit}]=E[(X'X)^{-1}X'y]=E[(X'X)^{-1}X'(X\beta+\epsilon)] $$

I interpret the first as what we imagine to be the expected value of $b_{ofit}$, given our overfitted model, but the second is the true expected value of $b_{ofit}$.

Can someone explain what's happening here? Should I assume that the two are equal, i.e. that $\gamma$ has zero expectation?

  • $\begingroup$ I gave a proof below that skips some steps in the algebra. If you want me to add more steps, let me know. $\endgroup$ – dlnB Mar 23 '19 at 23:09
  • 1
    $\begingroup$ Orthogonality of $X$ and $Z$ is only important if we estimate $y=X \beta +\epsilon$ when the true model is $y=X \beta+Z \gamma +\epsilon$. If $X$ and $Z$ are not orthogonal, then we have omitted variable bias. He is asking about the reverse case, i.e. when we estimate $y=X \beta+Z \gamma +\epsilon$, but the true model is $y=X \beta +\epsilon$, in which case $\hat{\beta}$ is still unbiased, but is inefficient. $\endgroup$ – dlnB Mar 24 '19 at 0:15

The two expressions you write both follow the wrong logic for the problem you describe. The first tells us what happens if we estimate $y_i=X_i \beta +\epsilon_i$ when the true model is $y_i=X_i\beta + Z_i \gamma + \epsilon_i$, as you replaced $y$ with the latter, but your regressor matrix is still just $X$. The second approach is estimating the true model correctly.

Correct approach: Notice that our regressor matrix is $[X Z],$ while the true value of $y$ is given by $y_i=X_i \beta +\epsilon_i$. Therefore: $$\left[ \begin{array}{cc} \hat{\beta}_{overfit}\\ \hat{\gamma} \end{array} \right]=\left[ \begin{array}{cc} X'X& X'Z \\ Z'X & Z'Z \end{array} \right] ^{-1} \left[ \begin{array}{cc} X'\\ Z'\end{array} \right](X\beta+\epsilon) = \left[ \begin{array}{cc} X'X& X'Z \\ Z'X & Z'Z \end{array} \right] ^{-1} \left[ \begin{array}{cc} X'(X\beta+\epsilon)\\ Z'(X\beta+\epsilon) \end{array} \right].$$

If we write $$\left[ \begin{array}{cc} X'X& X'Z \\ Z'X & Z'Z \end{array} \right] ^{-1}=\left[ \begin{array}{cc} A & B \\ C & D \end{array} \right],$$ it follows $$\hat{\beta}_{overfit}=AX'(X\beta+\epsilon)+BZ'(X\beta+\epsilon).$$

Taking expectations, and using the fact that $E(\epsilon|X,Z)=0$, we get $$E(\hat{\beta}_{overfit})=AX'X\beta+BZ'X\beta.$$ Using the rules of block matrix inversion, we get $$A=(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}$$ $$B=-(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}X'Z(Z'Z)^{-1}Z'.$$ It follows $$E(\hat{\beta}_{overfit})=(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}X'X\beta -(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}X'Z(Z'Z)^{-1}Z'X\beta.$$ Combining terms, we get $$E(\hat{\beta}_{overfit})=(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}(X'X-X'Z(Z'Z)^{-1}Z'X)\beta.$$ Using the fact that $(X'X-X'Z(Z'Z)^{-1}Z'X)^{-1}(X'X-X'Z(Z'Z)^{-1}Z'X)$ is simply an identity matrix, we get $$E(\hat{\beta}_{overfit})=\beta.$$

  • $\begingroup$ Do you use the Partitioned Regression Theorem to get the first expression for the estimators? $\endgroup$ – David Mar 24 '19 at 11:15
  • $\begingroup$ Nvm it's just from the normal equation. $\endgroup$ – David Mar 24 '19 at 12:38
  • $\begingroup$ Yeah the starting equation for $\left[ \begin{array}{cc} \hat{\beta}_{overfit}\\ \hat{\gamma} \end{array} \right]$ comes from the OLS formula with the usual $X$ replaced with $[X Z]$ and $y$ replaced with $(X \beta + \epsilon)$. If you have any other questions I'm happy to answer them. $\endgroup$ – dlnB Mar 24 '19 at 14:28
  • $\begingroup$ Intuitively you could say that the overfitted model is like fitting a "true" model with $\gamma = 0$. $\endgroup$ – Sextus Empiricus Apr 4 '19 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.