I have a data in which one group of observations (non-experienced users) may have different patterns of behavior (most of them are not captured by my data) from the other group's observations (experienced users). I am estimating the effect of another variable on a dependent variable, in a linear regression. I of course use the IsExpereincedUser binary variable as one of my controls. My question is - is it logical to cluster the errors around this variable? (as I believe that the unobserved patterns are correlated within each group), or should clustered robust error handling should be used only with multiple levels of the specific variable? Any reference would be also very helpful.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Cluster robust error theory is built on assumptions of large numbers of clusters with small numbers of observations. See Cameron Trivedi (2005) (Microeconomics: Methods and Applications) on p.834. Thus, your situation is not the ideal for clustered errors, and in particular, you should be cautious about interpreting the precise values of clustered errors that you get. (My guess is that probably the clustered errors would be too small). Nevertheless, you are definitely right to flag this as a potential issue and seek ways to address it. My overall take is that if there is a big difference between clustered and regular errors here, it means that you have model mis-specification problems. In that case, the clustered errors will not fix them! Instead, they simply help you to diagnose them in the first place.
One of the key things you should do here is to think about what might be the cause of correlation between the errors in the observations from the two groups. For instance, at a minimum, you may want to consider using not just your binary control for IsExperiencedUser, but also interacting that control with your other regressors. If any of your predictors have different effects on the two groups, then that would certainly lead to correlation in the errors. E.g. the true coefficient for $X$ on group 1 is 0.6 but it is $0.8$ for group 2. If the groups are equal size, then you'll get an estimate of $\hat{\beta}$ for that $X$ variable as $0.7$. This means that you'll tend to systematically overpredict $\hat{y}$ values for group 1 and systematically underpredict values for group 2. Here, you can fix the correlation of errors by including that interaction. But, the real reason that you include this interaction isn't to fix the correlation of the errors, it is because failing to do so means you estimate inaccurate parameters for each group!
A tougher situation will be if you have an omitted variable that for a given group is correlated with $X$. So, suppose your $X$ variable is a count for how many training sessions a user has attended. I'll assume that this is training to use some software program and your outcome is a measure of productivity. And you've got your two groups of experienced and inexperienced users. But, suppose that there is an omitted variable of whether the users have degrees in computer science. People who already have degrees in computer science are less likely to attend the training sessions, since they feel like they probably don't need them. And, experienced users are more likely to have computer science degrees. Now you have a situation in which you have an omitted variable (CS degree) which is correlated with your outcome (productivity) and with your predictor (# trainings attended). This is Bad! It will cause omitted variable bias.
In particular, when you go to predict productivity, you'll get estimates that are systematically too low for users with the CS degree you can't see and these people will be more likely to be experienced users, meaning that you'll get a correlation of prediction errors for people in that group.
In this case though, you need to find some way to solve this omitted variable problem - either by getting data on the omitted variable or by finding some instrumental variable strategy. Otherwise, you'll get a systematically biased estimate on the effect of your training program (in this case, you'll get a downwards bias on the estimate of its effectiveness).
So again, the discrepancy between the regular errors and the clustered errors here serves as a canary in the coal mine, rather than something that can be "corrected" by just switching your software specification to a different error calculation method.
The more typical situation where clustered errors can genuinely solve a problem is where it is more plausible that the source of the clustering is genuinely independent of your predictors. E.g. you're measuring the effects of a government program on health outcomes in different cities, but maybe some cities got hit with heat waves that hurt those health outcomes for all the observations of individuals in those cities, but there's no plausible reason those heat waves would be correlated with the implementation of the government program. It could be that if you see your clustered errors quite different from standard ones here, and can't fix it e.g. with interactions, that you can come up with a credible explanation like this for why the source of the clustering isn't correlated with your predictors. But, it'll be on you to do that, and to convince your audience that is actually what is going on.
