When to use fixed effects vs using cluster SEs?

Question

Suppose you have a single cross-section of data where individuals are located within groups (e.g. students within schools) and you wish to estimate a model of the form Y_i = a + B*X_i where X is a vector of individual level characteristics and a a constant.

In this case, suppose unobserved between-group heterogeneity biases your point estimates of B and their SEs since it is correlated with your independent variable of interest.

One option is to cluster your SEs by groups (schools). Another is to include group FEs. Another is to use both. What should one consider when choosing between these options? It is particularly unclear why one might cluster SEs by group AND use group FE. In my specific case, I have 35 groups and 5,000 individuals nested within each group. I have followed the discussion in this pdf, but it is not very clear on why and when one might use both clustered SEs and fixed effects.

(Please discuss the pros and cons of clustered SEs vs. FEs instead of suggesting I just fit a multilevel model.)

Answer 1

Both approaches, using group fixed effects and/or cluster-adjusted standard error take into account different issues related to clustered (or panel) data and I would clearly view them as distinct approaches. Often you want to use both of them:

First of all, cluster-adjusted standard error account for within-cluster correlation or heteroscedasticity which the fixed-effects estimator does not take into account unless you are willing to make further assumptions, see the Imbens and Wooldridge lecture slides for an good discussion of short and long panels and various issues related to this problem. There is also a novel paper about this topic by Cameron and Miller: A Practitioner's Guide to Cluster-Robust Inference which might be interesting for you. If you do not want to model the variance-covariance matrix and you suspect that within-cluster correlation is present, I advise to use cluster robust standard error because the bias in your SE may be severe (much more problematic than for heteroscedasticity, see Angrist & Pischke Chapter III.8 for a discussion of this topic. But you need enough cluster (Angrist and Pischke say 40-50 as a role of thumb). Cluster-adjusted standard error take into account standard error but leave your point estimates unchanged (standard error will usually go up)!

Fixed-effects estimation takes into account unobserved time-invariant heterogeneity (as you mentioned). This can be good or bad: On the hand, you need less assumptions to get consistent estimations. On the other hand, you throw away a lot of variance which might be useful. Some people like Andrew Gelman prefer hierarchical modeling to fixed effects but here opinions differ. Fixed-effects estimation will change both, point and interval estimates (also here standard error will usually be higher).

So to sum up: Cluster-robust standard error are an easy way to account for possible issues related to clustered data if you do not want to bother with modeling inter- and intra-cluster correlation (and there are enough clusters available). Fixed-effects estimation will take use only certain variation, so it depends on your model whether you want to make estimates based on less variation or not. But without further assumptions fixed-effects estimation will not take care of the problems related to intra-cluster correlation for the variance matrix. Neither will cluster-robust standard error take into account problems related to the use of fixed-effects estimation.

Answer 2

Fixed effects are for removing unobserved heterogeneity BETWEEN different groups in your data.

I disagree with the implication in the accepted response that the decision to use a FE model will depend on whether you want to use "less variation or not". If your dependent variable is affected by unobservable variables that systematically vary across groups in your panel, then the coefficient on any variable that is correlated with this variation will be biased. Unless your X variables have been randomly assigned (and they never will be with observation data), it is usually fairly easy to make the argument for omitted variables bias. You may be able to control for some of the omitted variables with a good list of control variables, but if strong identification is your number 1 goal, even an extensive list of controls can leave room for critical readers to doubt your results. In these cases, it is usually a good idea to use a fixed-effects model.

Clustered standard errors are for accounting for situations where observations WITHIN each group are not i.i.d. (independently and identically distributed).

A classic example is if you have many observations for a panel of firms across time. You can account for firm-level fixed effects, but there still may be some unexplained variation in your dependent variable that is correlated across time. In general, when working with time-series data, it is usually safe to assume temporal serial correlation in the error terms within your groups. These situations are the most obvious use-cases for clustered SEs.

Some illustrative examples:

If you have experimental data where you assign treatments randomly, but make repeated observations for each individual/group over time, you would be justified in omitting fixed effects, but would want to cluster your SEs.

Alternatively, if you have many observations per group for non-experimental data, but each within-group observation can be considered as an i.i.d. draw from their larger group (e.g., you have observations from many schools, but each group is a randomly drawn subset of students from their school), you would want to include fixed effects but would not need clustered SEs.

Answer 3

These answers are fine, but the most recent and best answer is provided by Abadie et al. (2019) "When Should You Adjust Standard Errors for Clustering?" With fixed effects, a main reason to cluster is you have heterogeneity in treatment effects across the clusters. There are other reasons, for example if the clusters (e.g. firms, countries) are a subset of the clusters in the population (about which you are inferring). Clustering is a design issue is the main message of the paper. Don't do it blindly.

Answer 4

@Kishore Gawande referenced the NBER working paper by Alberto Abadie, Susan Athey, Guido W. Imbens, and Jeffrey Wooldridge but I think it would be useful to repeat the key conclusions here as (per my reading) they do not necessarily align with every aspect of the the most accepted answers here.

First, clustered standard errors are a design rather than a model issue. Just because clustering standard errors makes a difference (results in larger standard errors than robust standard errors) is no reason that you should do it. Here's the top line: you should use clustered standard errors if you're working with a cluster sample or with an experiment where assignments have been clustered.

There's one exception. If there's no heterogeneity in the treatment effects and assignments have not been clustered, you don't have to use clustered standard errors. If you're using fixed effects, this requirement is looser. If there's no heterogeneity in the treatment effects, you don't have to use clustered standard errors. However, as Abadie et al. note, it's very unlikely that in practice there will be no heterogeneity in treatment effects, so this difference doesn't make much difference in practice. Hence, whether you're using fixed effects or not, if you're working with a cluster sample or clustered assignments, use clustered standard errors.

To quote Abadie et al. directly:

Without fixed effects, one should cluster if either (i) both $P_{C_n}$ < 1 (clustering in the sampling) and there is heterogeneity in the treatment effects, or (ii) σ2 > 0 (clustering in the assignment). With fixed effects, one should cluster if either (i) both PCn < 1 (clustering in the sampling) and there is heterogeneity in the treatment effects, or (ii) σ2 > 0 (clustering in the assignment) and there is heterogeneity in the treatment effects. In other words, heterogeneity in the treatment effects is now a requirement for clustering adjustments to be necessary, and beyond that, either clustering in sampling or assignment makes the adjustments important

In his answer, @Alex's says "Clustered standard errors are for accounting for situations where observations WITHIN each group are not i.i.d. (independently and identically distributed)" and provides the following example:

Alternatively, if you have many observations per group for non-experimental data, but each within-group observation can be considered as an i.i.d. draw from their larger group (e.g., you have observations from many schools, but each group is a randomly drawn subset of students from their school), you would want to include fixed effects but would not need clustered SEs.

This is misleading. If the sample is clustered and there is heterogeneity in the treatment effects (and there usually is), you need clustered standard errors.

To put this in the terms of survey sampling, if the design effect is greater than 1, i.e. observations from a group are not independent because they are more similar to each other than to observations from other groups, then you have to account for this. Ensuring that you sample from each group (e.g. school) is random, doesn't in any way bale you out here. Your effective sample size is less than your actual sample size. This is what leads to standard errors that are too narrow unless they are adjusted (via clustered standard errors) to account for this.

For all the fine print, include simulations and mathematical proofs, see Abadie et al. When Should You Adjust Standard Errors for Clustering.