Is it always better to analyze at the most granular level possible? What is the best unit of analysis in the context of hierarchically clustered data?

Question

Let's say that I have experimental data where the level of treatment is at a higher level of aggregation than the level of observation. For example, imagine some subset of schools adopted a new pedagogy for math class while the available data includes math test scores for all students across all schools.

Assuming that there is no spillage of the treatment effect and no interactions across schools, would using students or school-level aggregates maximize sensitivity with respect to detecting the treatment effect? Would there even be a meaningful difference?

My first instinct is to analyze at the lowest-level possible to maximize the sample size. However, I have been advised that the bottle-neck limiting the signal-to-noise ratio is at the level of treatment, and that if I were to use school-level aggregates (n=30) as opposed to students (n=10,000), there would be no real impact on the signal-to-noise ratio.

Can anyone please advise on the most desirable level of analysis for estimating the treatment effect in the context of hierarchically clustered data?

Answer 1

You don't have to choose! You can (and should) analyze the data at BOTH levels. This whole situation is one of the reasons that "multilevel modeling" (or "hierarchical modeling") was invented.

Your dependent variable is clearly at the student level, while the treatment is clearly at the school level, and students are "clustered" or "nested" within schools. So on one hand, if you try to "pretend" that your data is only at the student level you will be violating various assumptions about (for example) the independence of the error terms.

On the other hand you probably don't want to only analyze the treatment at the school level, since then you are no longer analyzing the treatment's effects on students, but instead just the effect it has on the average math score for all students at the school.

But this is what multilevel modeling is for. There are many ways to do this (and multiple ways to talk about or write the same kind of model) but I'll just give a simple example of a random intercept model that you might use to analyze these data. In this model "i" indexes students and "j" indexes schools

$TestScore_{ij} =\beta_0 + \zeta_j + \beta_1MathProgram_j+\epsilon_{ij}$

This is basically a linear regression model with math program (school level) as the independent variable and test score (student level) as the dependent variable, but in addition to the normal intercept $\beta_0$ we also have a school level random effect $\zeta_j $. We assume these J "school level deviations" are normally distributed around $\beta_0$. In other words we assume that students at the same school share an intercept and that these school level intercepts are normally distributed around some overall intercept. The model doesn't estimate the $\zeta_j $ values directly but it does use the data to estimate the variance of that normal distribution. This approach deals with the various violations of regression assumptions that come from having data at multiple levels. And unlike just adding a set of dummy variables for each school, it allows you to include other school level variables in the model (like the treatment variable). Of course, this model is still vulnerable to omitted variable bias at both the school and student level. So unless the treatment was randomly assigned to specific schools, you should consider adding other school level control variables to the model.

This is just a taste of the basic approach, but if you are interested in data like this you should read more about multilevel models, or hierarchical models (people use different terms to describe the same thing). What I've shown here is just a very simple example.

Answer 2

For a given data set, you don't actually have any choice about this - if you want to estimate a treatment effect, you can only analyze at the level that the treatment was applied. Analyzing at any other level fails to control alpha, so any standard errors and p-values you calculate would be meaningless.

Let's use your example, which seems to have a structure like this:

$$ Treatment -> Schools -> Students $$

This schematic reflects the reality of the structure of the data set, you can't decide to analyze it like this (or any other way):

$$ Schools -> Treatment -> Students $$

It's worth thinking about why the design of the experiment mandates a given analysis. Each school chooses to adopt the pedagogy at the school level, but there is likely random school-to-school variation in how the teachers are instructed to carry out the new lesson plans, etc. In fact, that's why we look at multiple schools - because there is random school-to-school variation in the implementation of the policy. However, every student at a given school experiences the same random error (so really, at the student level, any variation in treatment effect is really a systematic error), so you can't consider students to be IID samples. Another fairly obvious way to think about it is this - you would expect two students at the same school to be more similar than two students at different schools.

There is one thing to consider for this specific example, however. Let's say that you could somehow be sure that every school is implementing exactly the same strategy (so there is no school-to-school variation). You still could not choose to analyze at the level of students, however, because there's no way to ensure there is no classroom-to-classroom variation (after all, everyone in the same classroom is taught by the same teacher, who certainly plays a key role in pedagogy). So you could choose to analyze your data like this:

$$ Treatment -> Classrooms -> Students $$

This increases your effective sample size, which increases the precision of your estimate of treatment effect, but it relies on the assurance that there is no school-to-school variation in the treatment effect. As there is typically no way to ensure that, it's safer to stick with what you know - the treatment was applied to schools, so you have to perform an analysis that clusters at the "school" level.

The take-home message is this: the structure of your experiment justifies a specific analysis plan, you can't do something else without inflating the false positive rate or making strong (and typically unverifiable) assumptions.