I'm running an experiment where some of the users are shown a new widget when they become eligible (some conditions that are unrelated to the experiment itself). I have a control counterfactual group for whom I record when they are eligible for the new widget on a given visit, but don't show the widget. As a result I have data for both control and treated groups which tracks when they first saw (or could have seen) the widget and their behaviour before and after.
So data comes in a form:
userid | state | arm | date | visited 123 | pre | control | 11.10.18 | 1 123 | post | control | 12.10.18 | 1
for a user whos first (counterfactual) exposure to the widget happened on 12.10.18
Assuming for simplicity that I want to measure effect of this new widget on daily active usage of the app (# of visited from the table above), how should I go about that?
The approach that was more or less clear to me is to compute the metric of interest in both groups after the enrollment (after the moment they first saw the widget) and then run some statistical hypothesis test on both sides of the data. This doesn't really cover though the fact that the metric is computed over users who have different exposure time to the feature and overestimates the effect of people who are exposed the longest.
I was thinking of some sort of propensity score matching solution (like MatchIt or CausalImpact), but I don't understand how exactly to apply it: use enrollmenet day as a parameter?