1
$\begingroup$

My question is about RCTs where we receive many repeated observations per unit, but we don't necessarily want to aggregate them together. A concrete example is as follows.

Suppose I am running an RCT for two versions of a studying tool for students in the real world. Each student reports back their grades on homework assignments, but we don't know in advance how many observations they will report.

The hypothesis is that the treatment group will increase their homework scores (say, from 70% to 75%). I have student covariates at the time of treatment assignment (e.g., GPA, year in school, etc), which I plan to use in the regression.

Treatment Student id Observations
1 1 90/100 (subject 1), 8/10 (subject 3), 7/10 (subject 5)
1 2 80/100 (subject 2)
1 3 40/50 (subject 4), 15/20 (subject 2)
0 4 70/100 (subject 2)
0 5 7/10 (subject 1), 40/50 (subject 3)
0 6 7/10 (subject 5)

A few specific questions:

  • One way to do this is to aggregating all of the data together by summing numerator and denominator. Now each student has a single observation (for student 1, it'd be 105/120), but this seems like I wouldn't be missing the full information I have. Is there a way to design a regression setup that can help me analyze this data without aggregating? Should I use student random effects in addition to the student covariates?
  • Suppose the students also report the subjects that the homeworks were for. Since these are observed after treatment assignment, they shouldn't be used as covariates in a regression model. Is there any way to use this information? Because the observations can be for different subjects, there will be some variation in the observations which I'm not sure how to handle.
Improve this question
$\endgroup$
2
  • $\begingroup$ I'm confused as to why the homework variable is random in the first place. Also which variable is your dependent variable here and what is your hypothesis? That may make it easier for others to answer. $\endgroup$
    – Shawn Hemelstrand
    Commented Nov 7 at 11:07
  • $\begingroup$ @ShawnHemelstrand Thanks for the suggestion! I've edited the question. The dependent variable is the fraction of correct answers for a given student. My hypothesis is that the treatment group has a higher fraction of correct answers. $\endgroup$
    – gaussiandynamics
    Commented Nov 8 at 13:29

1 Answer 1

Reset to default
1
$\begingroup$

I would regress scores on a binary treatment flag, a set of binary flags for each subject (dropping one), interactions between treatment and included subject flags, plus a constant. I would cluster the standard errors by student since that is the level at which treatment is assigned. Pre-assignment covariates are good for improving power or possibly testing for prespecified heterogeneous effects (giving out calculators may only help STEM majors). I would then test the joint null that the effects for each subject are less than or equal to zero against greater.

This approach assumes there are no spillovers from treated students to control pupils. There are many ways this could be violated. For instance, if control students get help from treated students, that would attenuate the effect.

If you think students select courses based on treatment (e.g., taking more challenging courses when treated), that requires more thought. But if they sign up before randomization and don't switch after, but you only find out the course later, that's not an issue.

It's not clear if you see all the assignments or just some. It could be problematic if students report selectively or don't complete them.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you @dimitriy for always being a great source of knowledge on this site. Appreciate it. A few follow-ups if you don't mind: "I would then test the joint null that the effects for each subject are less than or equal to zero against greater." I didn't quite understand what you meant here. Maybe you are talking about a the situation where we are using interaction terms to test the HTEs? (treatment : subject) $\endgroup$
    – gaussiandynamics
    Commented Nov 8 at 18:36
  • $\begingroup$ "But if they sign up before randomization and don't switch after, but you only find out the course later, that's not an issue." What if the sequence of events is: (1) randomization, (2) students randomly decide on courses, but not affected by treatment? Naively, since this is a random outcome that happens after treatment, can it be a covariate in the regression? $\endgroup$
    – gaussiandynamics
    Commented Nov 8 at 18:36
  • $\begingroup$ @gaussiandynamics The effect in the specification I proposed is the coefficient on the treatment dummy plus the coefficient on the treatment x subject interaction. This allows the effect to vary by subject, so you will have 5 different effects. For example, a calculator treatment may have a different effect for math than literature. $\endgroup$
    – dimitriy
    Commented Nov 8 at 20:51
  • $\begingroup$ If class selection is not influenced by treatment, I don't see a problem. Including subject effects allows you to have different baselines. For instance, some subjects may be harder than others, the grades may be more compressed, or one teacher may mumble. So, subjects may have some independent influence on the scores. $\endgroup$
    – dimitriy
    Commented Nov 8 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.