Causal Inference - Revenue Per User

Question

Let's say I'm running an A/B test where I randomly select users to be in the control group (group A) or the treatment group (group B). I want to measure the impact of the treatment on the revenue per user for my video game. However, I know that some users are heavy gamers (20+ hours a week) while some users are very casual (< 3 hours a week). How can I control for this when estimating the impact of the treatment on revenue per user? As an extra note, the experiment has already started and the users have already been randomly selected into the groups.

I was thinking that I could run a linear regression model where the outcome is the revenue per user and the control variables would be a treatment indicator and the average hours played per week. Is this the right approach?

Answer 1

First of all, there is no need for you to control for average hours played. If you run a simple regression just with the treatment indicator, this will give you an estimate of what the effect of your treatment will be - averaged across the distribution of average hours played in your sample.

However, it may be a good idea to take into account average hours played. Then, you need to make sure that this measure is not taken after the treatment was administered, but rather before. Otherwise, it may be affected by the treatment itself, and controlling for average hours played may introduce post-treatment bias (see this answer).

If you make sure that this is not the case, a viable approach would be to run a regression of the outcome on the treatment indicator, average hours played, and an interaction between the two. As pointed out by Noah, mean-center average hours played. Then, the coefficient on your treatment can still be directly interpreted as the average treatment effect.

Furthermore, the coefficient on average hours played and on the interaction term could then inform your decision to perhaps implement the treatment not for everyone, but just a subgroup of players. Also, efficiency is likely to increase, and your estimate of the treatment impact will be more precise.