Skip to main content
edited tags
Link
usεr11852
  • 46.1k
  • 3
  • 107
  • 166
Notice removed Current answers are outdated by Parseltongue
Bounty Ended with Sextus Empiricus's answer chosen by Parseltongue
added 177 characters in body
Source Link
Parseltongue
  • 1.1k
  • 2
  • 11
  • 28

I have data from an experiment I ran in which I paired individuals up to play a game with another person. Before and after the game, some baseline and endline measures are collected, and the DV of interest is usually some change score on these measures. The "treatment" is whether a given participant played with someone of the opposite sex. Crucially, I cluster standard errors at the "team" level to control for within-team correlations.

We find that the game is effective at changing the DVs of interest. However, we really want to know what about the game is driving the change. There are a huge number of potential demographic moderators (age, income, education, political affiliation, etc) that might generate heterogeneous treatment effects, and there are also a lot of "in-game" variables that might matter (e.g. the total amount of chatting participants did with one another).

I'm looking for an approach that allows me to pursue exploratory moderator analysis in this way with interpretable results.

I came across the grf package which can construct causal random forests, but the package seems extremely limited in its ability to generate interpretable output. I'm also skeptical of how accurate its estimates are in light of some recent papers showing that such models underperform relative to Bayesian approaches (see here).

I came across the causalml Python package, which has implementations of various "meta-learners" and uplift algorithms, which seem promising, but I don't know enough about this literature to know whether it resolves my problem.

Ideally, I'd like to be able to say what moderators are most important at producing "lift" in the treatment effectIdeally, I'd like to be able to say what moderators are most important at producing "lift" in the treatment effect, along with some precise quantification of what that lift entails. Furthermore, along withmy specific use-case has some precise quantification of whatclustering that lift entailsI normally account for via cluster-robust standard errors, so if possible, the solution should account for this.

Any advice?

I have data from an experiment I ran in which I paired individuals up to play a game with another person. Before and after the game, some baseline and endline measures are collected, and the DV of interest is usually some change score on these measures. The "treatment" is whether a given participant played with someone of the opposite sex. Crucially, I cluster standard errors at the "team" level to control for within-team correlations.

We find that the game is effective at changing the DVs of interest. However, we really want to know what about the game is driving the change. There are a huge number of potential demographic moderators (age, income, education, political affiliation, etc) that might generate heterogeneous treatment effects, and there are also a lot of "in-game" variables that might matter (e.g. the total amount of chatting participants did with one another).

I'm looking for an approach that allows me to pursue exploratory moderator analysis in this way with interpretable results.

I came across the grf package which can construct causal random forests, but the package seems extremely limited in its ability to generate interpretable output. I'm also skeptical of how accurate its estimates are in light of some recent papers showing that such models underperform relative to Bayesian approaches (see here).

I came across the causalml Python package, which has implementations of various "meta-learners" and uplift algorithms, which seem promising, but I don't know enough about this literature to know whether it resolves my problem.

Ideally, I'd like to be able to say what moderators are most important at producing "lift" in the treatment effect, along with some precise quantification of what that lift entails.

Any advice?

I have data from an experiment I ran in which I paired individuals up to play a game with another person. Before and after the game, some baseline and endline measures are collected, and the DV of interest is usually some change score on these measures. The "treatment" is whether a given participant played with someone of the opposite sex. Crucially, I cluster standard errors at the "team" level to control for within-team correlations.

We find that the game is effective at changing the DVs of interest. However, we really want to know what about the game is driving the change. There are a huge number of potential demographic moderators (age, income, education, political affiliation, etc) that might generate heterogeneous treatment effects, and there are also a lot of "in-game" variables that might matter (e.g. the total amount of chatting participants did with one another).

I'm looking for an approach that allows me to pursue exploratory moderator analysis in this way with interpretable results.

I came across the grf package which can construct causal random forests, but the package seems extremely limited in its ability to generate interpretable output. I'm also skeptical of how accurate its estimates are in light of some recent papers showing that such models underperform relative to Bayesian approaches (see here).

I came across the causalml Python package, which has implementations of various "meta-learners" and uplift algorithms, which seem promising, but I don't know enough about this literature to know whether it resolves my problem.

Ideally, I'd like to be able to say what moderators are most important at producing "lift" in the treatment effect, along with some precise quantification of what that lift entails. Furthermore, my specific use-case has some clustering that I normally account for via cluster-robust standard errors, so if possible, the solution should account for this.

Any advice?

Notice added Current answers are outdated by Parseltongue
Bounty Started worth 100 reputation by Parseltongue
Source Link
Parseltongue
  • 1.1k
  • 2
  • 11
  • 28

Heterogeneous Treatment Effects - Interpretable Methods?

I have data from an experiment I ran in which I paired individuals up to play a game with another person. Before and after the game, some baseline and endline measures are collected, and the DV of interest is usually some change score on these measures. The "treatment" is whether a given participant played with someone of the opposite sex. Crucially, I cluster standard errors at the "team" level to control for within-team correlations.

We find that the game is effective at changing the DVs of interest. However, we really want to know what about the game is driving the change. There are a huge number of potential demographic moderators (age, income, education, political affiliation, etc) that might generate heterogeneous treatment effects, and there are also a lot of "in-game" variables that might matter (e.g. the total amount of chatting participants did with one another).

I'm looking for an approach that allows me to pursue exploratory moderator analysis in this way with interpretable results.

I came across the grf package which can construct causal random forests, but the package seems extremely limited in its ability to generate interpretable output. I'm also skeptical of how accurate its estimates are in light of some recent papers showing that such models underperform relative to Bayesian approaches (see here).

I came across the causalml Python package, which has implementations of various "meta-learners" and uplift algorithms, which seem promising, but I don't know enough about this literature to know whether it resolves my problem.

Ideally, I'd like to be able to say what moderators are most important at producing "lift" in the treatment effect, along with some precise quantification of what that lift entails.

Any advice?