Determine if treatment effect is distributed equally across participants in a study or driven by a few participants

Question

This is perhaps a malformed question, but I'll give it a shot. Let's say I have a pre and post measure of some outcome variable and a dummy variable treatment that is 1 if participants are in a treatment condition and 0 if control.

Let's say I model the effect on the outcome variable simply with:

lm(outcome_post - outcome_pre ~ treatment)

Let's say the coefficient on treatment is 3.0 and is highly statistically significant. What I want to answer is: how evenly distributed is this treatment effect across participants? It could be the case that only a handful of participants benefits from the treatment, but they benefit to a significant degree.

Is there a test or procedure I can use to test whether treatment effects are evenly distributed in the sample?

Answer 1

This can be addressed by reframing your model - which is worth doing anyway.

Change scores like [outcome_post - outcome_pre] are not the recommended approach to model pre/post data. Instead, a better approach is to use the pre measurement as a covariate i.e. do lm(outcome_post ~ outcome_pre * treatment). The interaction term will then tell you what you want to know about how variable the treatment effects are.

This is an excellent thread that explains how to analyse pre-post designs in more detail, though it doesn't explicitly address your question about variability in treatment effects: Best practice when analysing pre-post treatment-control designs

Answer 2

Apart from the issue of pre/post treatment issue, regarding the influence of specific points, I would recommend to look into Cook's distance, a metric that allows you to check if there are / which are the influential observations in your model.

I know that check_model() function from the performance package produces a plot that can indeed identify these samples. But, I would recommend following the approach presented here for the olsrr package: https://cran.r-project.org/web/packages/olsrr/vignettes/influence_measures.html. I would say it can indeed give you the tools to ascertain if the treatment effect is homogeneous or not.