2
$\begingroup$

Let's say I'm running an A/B test where I randomly select users to be in the control group (group A) or the treatment group (group B). I want to measure the impact of the treatment on the revenue per user for my video game. However, I know that some users are heavy gamers (20+ hours a week) while some users are very casual (< 3 hours a week). How can I control for this when estimating the impact of the treatment on revenue per user? As an extra note, the experiment has already started and the users have already been randomly selected into the groups.

I was thinking that I could run a linear regression model where the outcome is the revenue per user and the control variables would be a treatment indicator and the average hours played per week. Is this the right approach?

$\endgroup$
3
$\begingroup$

First of all, there is no need for you to control for average hours played. If you run a simple regression just with the treatment indicator, this will give you an estimate of what the effect of your treatment will be - averaged across the distribution of average hours played in your sample.

However, it may be a good idea to take into account average hours played. Then, you need to make sure that this measure is not taken after the treatment was administered, but rather before. Otherwise, it may be affected by the treatment itself, and controlling for average hours played may introduce post-treatment bias (see this answer).

If you make sure that this is not the case, a viable approach would be to run a regression of the outcome on the treatment indicator, average hours played, and an interaction between the two. As pointed out by Noah, mean-center average hours played. Then, the coefficient on your treatment can still be directly interpreted as the average treatment effect.

Furthermore, the coefficient on average hours played and on the interaction term could then inform your decision to perhaps implement the treatment not for everyone, but just a subgroup of players. Also, efficiency is likely to increase, and your estimate of the treatment impact will be more precise.

$\endgroup$
  • 2
    $\begingroup$ +1. As a note, if you center your covariate before including it in the model and interaction, the coefficient on the treatment is the average causal effect. If you don't center it, the coefficient is the effect when the covariate is equal to 0. $\endgroup$ – Noah Jun 29 '18 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.