I made up an example so as to illustrate my question with some more context. Say there are two national parks, and a ranger is interested in finding out how the number of rabbits (Y) varies with the number of wolves (X), so the ranger collects a dataset {(Xi,Yi),i=1..N} over some period. However, unknown to the ranger, there is another variable, C, the rain-fall, that affects the rabbit population (more rain more grass). Let's keep it simple, the rain-fall of national park A is always 100ml, and is always 10ml for the other parks. And the true (but unknown to ranger) relationship is Y = 0.01 * X + C + noise. If the ranger simply regresses Y on X, does she violates the assumption that the observations (Xi,Yi) are drawn i.i.d from the joint distribution of P(X,Y)?

The reason why I'm asking this is because I feel that the observations made within a park are correlated with each other. I also feel here's some confusion of concepts, but can't pinpoint which. So any explanation/clarification will be great.


1 Answer 1


Firstly, when doing regression (with OLS), we essentially split the variable of interest $y$ into a deterministic part $\beta_0 + \beta_1 X$ and a stochastic part $u$, i.e. the 'noise', as you also point out: $y=\beta_0 + \beta_1 X+u$. Thus we are interested in the distribution of the error terms, i.e. the noise $u$, since these capture the whole randomness. Hence the classical assumption of $iid$ is with respect to $u \sim iid$, not the set $\{y_i, X_i \}$.

Secondly, the important assumption needed in order to have a correctly specified model is exogeneity of the regressors, i.e. $E[u|X]=0$. If it holds that $u \sim iid$, then exogeneity is automatically implied. If you leave out a regressor variable $C$ and this variable depends (in the mean) on another regressor variable $X$, then the model based only on $X$ is wrongly specified and you have an omitted variable bias. This is because you implicitly 'force' $C$ into the noise term $u$. If $C$ depends on $X$ (in the mean), then obviously $u$ will depend on $X$ (in the mean) and it will not be $iid$ anymore, but rather depend on the value of $X$. Exogeneity will also not be given. If on the other hand $C$ affects (in the mean) $y$ but not $X$, then $C$ can be considered noise and captured in the noise variable $u$. Then the model based only on $X$ will be correctly specified and coefficients unbiased - except for the intercept. So, no harm in not knowing $C$ in this case, $u$ will still be $iid$, just maybe only not with zero mean (which will be capture by the intercept, making it biased). It just remains to think about whether $wolves$ and $rain$ are or not independent of each other.

Lastly, your data set here represents a time series. Since it is a birth/death process I highly doubt that it will be white noise, but it will rather have some seasonal effects, e.g. higher population in summer (due to births) and lower in winter. Thus there will most likely be serial correlation between the noise terms $u_i$, i.e. the rabbit population today is very likely very similar to the population yesterday. This would mean that the $iid$ assumption is not satisfied if we simply regress $y$ on $X$ and neither is exogeneity. Note that this is due to the nature of the stochastic process, not leaving $C$ out. You would have the same problem even if you would include $C$ in the model. This is because the regressor $X$ depends on time too, hence time essentially becomes an omitted variable and thus $X$ is not sufficient by itself. In this case you would have to make the data stationary (decompose the time series into trend, seasonality, noise, and fit the noise e.g. with an ARMA process. See Box-Jenkins method. Literature: Introduction to Time Series and Forecasting 3rd Ed- Brockwell & Davis)

To conclude: I suggest thinking in terms of exogeneity of $u$ rather than $u \sim iid$. If $u \sim iid$ doesn't hold, it's harder to prove a LLN and/or a CLT, but exogeneity sufficed for unbiasedness. Omitting a variable that affect the response but not the feature of interest only biases the intercept $\beta_0$, but not $\beta_1$. And when we're looking at time series, always check for seasonality or a trend or any non-stationary behavior, because if it is present, then $u$ is most definitely not $iid$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.