Relative novice here. I am running a regression in an observational setting in which Y is the outcome and D is the treatment indicator. Observations are drawn from 3 different geographic groups designated by X. Is the proper approach to:

  1. Regress Y on D and cluster the standard errors by group.
  2. Regress Y on X and D.
  3. Regress Y on X and D and cluster the standard errors by group.

When pursuing option #3 I am seeing much higher statistical significance -- and I'm worried somehow that including both dummies and the clustering in a cross-sectional setting is problematic.

In principle, what are the tradeoffs between the 3 approaches? Which is most likely to offer an unbiased estimate of the treatment effect D (assuming other covariates -- not included here -- are balanced between the 2 groups)?

  • $\begingroup$ I guess you meant, "regress $Y$ on $D$", etc. The dependent variable is regressed on a set of explanatory variables. $\endgroup$ – StasK Feb 1 '13 at 13:11

So with clustered standard errors in your situation you are saying, basically, that you are happy with the stability of the estimate of variance based on three observations, and equally happy to assume that 3 is infinity in terms of using asymptotic normality for your inference. See sec. 8.2.3 of Mostly Harmless Econometrics. 42 is sort of infinity; there is no freaking way 3 is. Moreover, your approach 3) should have broken down as you would not have any degrees of freedom left for clustered standard errors, having more regressors than clusters.

The only approach I would buy in your situation is regressing $Y$ on $D$ or $Y$ on $D \times X$. In the latter case, you would want to test both the main effect of the treatment and its interactions with the regions.

  • $\begingroup$ StasK - Original user here. So you're saying that if there are 100s of units per cluster, but only 3 clusters, there is no real way to account for different variances in each cluster? This is an observational study, so the number of clusters can't be increased. Regressing Y on D*X makes sense, but I also expect the untreated observations within each group to have different means... so isn't Y on D and X more appropriate? As a more concrete example, say that students in a school X were randomly exposed to one of two tests. The randomization was also conducted at 2 other schools. ... $\endgroup$ – reson Feb 1 '13 at 15:18
  • $\begingroup$ I would expect there to be school-level differences in the mean and variance on test scores - across both the treated and untreated groups. $\endgroup$ – reson Feb 1 '13 at 15:18

You should use fixed effects if I understand your example correctly: You are using a difference-in-difference approach, where the groups ("schools") have different averages to begin with (for unobserved reasons) and were not randomly selected. Using fixed effects in the regression gives the the model different y-intercepts for each "school" in your example. Essentially, you are assuming the treatment effect (slope / beta) is the same for all schools, but the mean score is different.

Clustering standard errors should be used when the standard errors are correlated within groups but not across groups. For example, a stratified sample where neighborhoods were randomly chosen and then students were randomly chosen within that neighborhood. We'd expect the treatment effect to be correlated within neighborhoods but not across neighborhoods.

In either case, clustering standard errors with only three clusters is not asymptotically valid, as pointed out above (you'd need at LEAST 42 clusters).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.