# OLS: $E[\epsilon_{it}^T\epsilon_{it}] \not= 0$ in 1st equation biases standard errors in 2nd equation?

Suppose ${X_{it}},{Y_{it}}$ are time series with $X_{it}\sim N(0.1,1)$, ($\sigma^2(Y_{it}) = 1$ and $mean(Y_{it})$ is similar to that for $X_{it}$, but changes when the dummy = 1). and $t \in \{1,2,...,200\}$, $i \in \{1,2,...,N\}$. In a real world setting this will be periodic stock market returns over $N$ firms (but you can ignore this). There is a dummy, $D_t$ that's equal to unity over $t \in \{150,151,...,200\}$ and equal to zero otherwise. The time series model to be estimated with OLS $\forall i$ is:

$(1) Y_{it} = \alpha_i + \beta_i X_{it} + \gamma_i D_{t} + \epsilon_{it}$

This model generally adheres to the Gauss-Markov assumptions for each $i$. However, we have $E[\epsilon_{it}^T \epsilon_{jt}] \not= 0$ for all $i$ and $j$.

The next step is to construct a vector of gammas using the $N$ estimates of model $(1)$. Call this vector $\bf{\hat{\gamma}}$. We then use this in the cross-sectional model:

$(2) \hat{\gamma}_i = a + b Z_i + u_i$

where $Z_i$ is some cross-sectional variable that doesn't cause any violations in OLS assumptions and is relevant for explaining $\hat{\gamma}_i$.

The claim in the applied econometrics literature is that $E[\epsilon_{it}^T \epsilon_{jt}] \not= 0$ in model $(1)$ leads to (i) No problem for the OLS coefficient estimates in $(2)$, but (ii) Biased standard errors in $(2)$.

• Can someone please post ideas about why this is the case?

• I don't understand what $\epsilon_{it}^T$ is in the expression $E[\epsilon_{it}^T \epsilon_{jt}] \not= 0$. Of course $\epsilon_{it}$ is a scalar and you can't transpose a scalar. This is seen HERE, where they apply this methodology.

-
You say that you don't understand why variance estimates are biased in eq 2 and then you say that we can ignore your $\gamma$ estimate which happens to be equation 2 ? I think I understand what do you mean and could give an speculative answer to that but it will be better for you to precise your question. –  JDav Aug 20 '12 at 11:33
In your setup $Y_{it}$ cannot be stationary, since its mean depends on $t$. –  mpiktas Aug 20 '12 at 13:45
There are three versions of the expectation (one in the title, another in the body, and a third in the comments). All of them incorporate a mysterious transpose even though in all the cases only scalars are present. Do you mind editing your post to clarify? –  cardinal Aug 20 '12 at 13:48
@mpiktas Correct observation, $Y_{it}$ has different mean after $t=150$ (given $\gamma_i \not= 0$). Thanks. –  user13253 Aug 20 '12 at 13:49
Some good answers have been delivered -- I would just add that this needs to be estimated as a random coefficients model (aka multilevel model for sociologists and psychologists, aka mixed model for biostatisticians). If economists do not know this, and estimate it with a two-step procedure, that's just too bad for them (and I am still waiting for Fama-Macbeth standard errors to die, which apparently they just don't want to do). –  StasK Aug 23 '12 at 15:20

In order to be sure you need to go into the details, this implies comparing the true variance covariance matrix with the one you get in the second ols stage.

The true one:

This can be obtained by replacing eq.2 into eq.1, pooled OLS follows, and from it, the true $\hat a , \hat b$ variance covariance matrix:

$Y_{it} = \alpha_i + \beta_i X_{it} + aD_t + bD_tZ_{i} +D_t u_{i} + \epsilon_{it}$

Using matrix notation to split the equation in $\gamma$ parameters and others leads to:

$Y = X\theta + Z\gamma + \varepsilon$

where we are interested in $V(\hat \gamma)$ , $\gamma=[a \; b]$, Z is a two column vector $Z=[D_t \; D_tZ_i]_{[i=1,..,N;t=1,...,T]}$ ( a similar structure defines X but this is not of interest) and where $V(\varepsilon) =\Sigma$ has a full structure of between firms covariances that's why it's not diagonal ($\sigma^2I_{NT}$) like in the GAUSS-MARKOV assumptions. By Frish-Waugh we can express $\gamma$ ols as :

$\hat \gamma = (Z'M_{X}Z)^{-1}Z'M_{X}Y$ where $M_X= I-X(X'X)^{-1}X'$

which implies the following true variance:

$V(\hat \gamma) = H\Sigma H'$ where $H = (Z'M_{X}Z)^{-1}Z'M_{X}$

The other one

Under the assumption of non correlated firms (and time periods but this is not the issue), $\Sigma$ has a simplier diagonal structure $\Delta$. This means that $\Delta$ triangular terms are 0. Under an even simpler specification,( the one that is estimated by default by econometric and statistical software for OLS) $\Sigma$ follows GAUSS-Markov assumptions meaning that even the diagonal terms are equal thus $\Sigma$ is downgraded to $\sigma^2I$

This implies that not considering between firms correlation would lead to $V(\hat\gamma)$as:

$V(\hat \gamma) = H\Delta H'$ or $V(\hat \gamma) = H\sigma^2I H' \equiv \sigma^2(Z'M_xZ)^{-1}$

which, as it can be seen, are not equal to the true one.

-
With different words.. I'm basiclly giving the same answer @mpiktas gave –  JDav Aug 22 '12 at 11:50
(1) Really fantastic. (2) Seems like you've ignored $D_i u_i$ when you expressed the model in matrix form? This shouldn't change anything you've done though. (3) Would you know why Portfolio OLS gives correct SEs? (See the 1986 paper I linked). Don't bother with answer (3) if you wouldn't enjoy figuring that problem out. –  user13253 Aug 22 '12 at 12:35
(2) I didn't put all the definitions in order to let some for intuition and to avoid kronecker products... this way the demo goes "faster". But you may deduce that the new random term is $\varepsilon_{it} = D_tu_i + \epsilon_{it}$, this means that if firms were correlated by $u_i$ then this causes the new random term $\varepsilon_{it}$ to be correlated on its firms dimension as well. (3) didn't hear about a portfolio OLS, but i guess it's a just another name to something that already exists in standard econometrics to del with full Var matrices, like WLS, or Robust OLS, etc. –  JDav Aug 22 '12 at 12:48
(3) a good estimate implies a good $\Sigma$ estimate , the portfolio OLS somehow is estimating the full structure $\Sigma$ and not only variances without covariances: $\Delta$ or a single variance: $\sigma^2$ –  JDav Aug 22 '12 at 12:58
I think the notation is innacurate, he uses scalars where vectors are needed to refer to the fact that between firms covariances are not zero so his notation implies $\epsilon_{it} = [\epsilon_{it}]_{i=1,...,N}$ is a N rows vector. Another interpretation is that he is refering to $ij$ element of $\epsilon_{.t}^T\epsilon_{.t}$. In both cases he means the same thing, but as it's not a quantitative journal ambiguities in math notation happen... –  JDav Aug 23 '12 at 12:43

I am putting up another answer with more details.

In standard linear regression model (in matrix form):

$$Y=X\beta+\varepsilon$$

the OLS estimate is the following

$$\hat\beta=(X^TX)^{-1}X^TY.$$

Its variance then is

$$Var(\hat\beta)=(X^TX)^{-1}X^TVar(Y)X(X^TX)^{-1}.$$

The usual assumption for regression is that

$$Var(Y)=\sigma^2I,$$

where $I$ is the identity matrix. Then

$$Var(\hat\beta)=\sigma^2(X^TX)^{-1}.$$

Now in your case you have two models:

$$Y_{i}=M_i\delta_i+\epsilon_i$$

and

$$\Gamma=Lc+u,$$

where

• $Y_i^T=(Y_{i1},...,Y_{iT})$,
• $M_i=[1,X_i,D]$, with $X_i^T=(X_{i1},...,X_{iT})$, $D^T=(D_1,...,D_T)$
• $\delta_i^T=(\alpha_i,\beta_i,\gamma_i)$
• $\epsilon_i^T=(\epsilon_{i1},...,\epsilon_{iT})$
• $\Gamma^T=(\gamma_1,...,\gamma_n)$
• $L=[1,Z]$, with $Z^T=(Z_1,...,Z_n)$
• $c^T=(a,b)$
• $u^T=(u_1,...,u_N)$.

Note that you state second model for the estimates of $\gamma$, which is not usual, hence I restate it in usual form, for the "true" $\gamma$.

Let us write down the covariance matrix for OLS estimates of coefficients $c$:

$$Var(\hat{c})=(L^TL)^{-1}L^TVar(\Gamma)L(L^TL)^{-1}$$

The problem is that we do not observe $\Gamma$. We observe the estimates $\hat\Gamma$. $\hat\gamma_i$ is part of vector

$$\hat\delta_i=\delta_i+(M_i^TM_i)^{-1}M_i^T\epsilon_i.$$

Assume that $\delta_i$ are random and independent with $\epsilon_i$ and $M_i$. This surely holds for $\gamma_i$ so we do not lose anything if we extend this for other elements of $\delta_i$.

Let us stack all $\hat\delta_i$ on top of each other:

$$\hat\delta^T=[\delta_1^T,...,\delta_N^T]$$

and explore the variance of $\hat\delta$:

$$Var(\hat\delta)=\begin{bmatrix} Var(\hat\delta_1) & cov(\hat\delta_1,\hat\delta_2) & \dots & cov(\hat\delta_1,\hat\delta_N)\\ \dots & \dots & \dots & \dots\\ cov(\hat\delta_n,\hat\delta_1) & cov(\hat\delta_n,\delta_2) & \dots & Var(\hat\delta_N) \end{bmatrix}$$

Assume that $Var(\epsilon_i)=\sigma^2_\epsilon I$ and that $E\epsilon_i\epsilon_j^T=0$. For $i\neq j$ we have

\begin{align} cov(\hat\delta_i,\hat\delta_j)&=cov(\delta_i,\delta_j)+cov((M_i^TM_i)^{-1}M_i^T\epsilon_i,(M_j^TM_j)^{-1}M_j^T\epsilon_j)\\ &=(M_i^TM_i)^{-1}M_i^TE(\epsilon_i\epsilon_j^T)M_j(M_j^TM_j)^{-1}\\ &=0 \end{align}

For diagonal elements we have

$$Var(\hat\delta_i)=Var(\delta_i)+\sigma_\epsilon^2(M_i^TM_i)^{-1}$$

Let us turn back to variance of $\hat c$. Since we substitute $\hat\Gamma$ instead of $\Gamma$ the variance is the following

$$Var(\hat{c})=(L^TL)^{-1}L^TVar(\hat\Gamma)L(L^TL)^{-1},$$

We can extract $Var(\hat\Gamma)$ from $Var(\hat\delta)$ by selecting appropriate elements:

$$Var(\hat\Gamma)=Var(\Gamma)+diag(g_1,...,g_n)$$

where $g_i$ is the element of $\sigma_\epsilon^2(M_i^TM_i)^{-1}$ corresponding to the $Var(\hat\gamma_i)$. Each $g_i$ is different from $g_j$ since they correspond to different $X_{it}$ and $X_{jt}$ which are not assumed to be equal.

So we get the surprising result, that algebraically even if we assume all the necessary properties, the resulting covariance matrix at least algebraically will not be equal to usual OLS covariance matrix, since for that we need that $Var(\hat\Gamma)$ is constant times identity matrix which it is clearly not.

All the formulas above were derived assuming that $X_{ij}$ are constant, so they are conditional on $X_{ij}$. This means that we actually calculated $Var(\hat\Gamma|X)$. By putting additional assumptions on $X_{ij}$, I think it would be possible to show that unconditional variance is OK.

The independence assumption placed on $\epsilon_i$ can also be relaxed to uncorrelatedness.

It would also be possible to use simulation study to see how covariance matrix differ if we use $\hat\Gamma$ instead of $\Gamma$.

-

I think the problem lies in definition of the second model. I think it is assumed that

$$\gamma_i=a+bZ_i+u_i$$

with the usual assumption that

$$cov(\gamma_i,\gamma_j|Z_1,...,Z_N)=0,$$

i.e. that the $\gamma_i$ are not correlated if we control for $Z_i$. Now when you substitute $\hat{\gamma}$ instead of $\gamma$, you need to check whether the assumption holds, i.e. if

$$cov(\hat{\gamma_i},\hat{\gamma}_j|Z_i)=0.$$

Now

$$\hat{\gamma}_i=\gamma_i+L(\epsilon_{it}),$$

where $L$ is some linear function. It is safe to assume that $\epsilon_{it}$ is independent of $Z_i$, but if $E\epsilon_{it}\epsilon_{jt}\neq0$, the necessary assumption does not hold.

Since the uncorrelatedness assumption is central to calculation of usual OLS statistics, this gives the reason why the standard errors are biased.

This was a rough outline, but I think the idea should work if you'd get into nitty gritty details of OLS machinery.

-