What to do if you Double count in experimental design?

Question

I have to run A/B/n tests for a subscription service. Generally computing metrics for this situtation is ok:

• For example, coversion rate experiments. We have 1000 prospects in a group, and (say) 89 get through the entire funnel to become customers. So the empirical conversion rate is 89/1000

Now suppose we are in the following situation:

A customer can perform an action multiple times during the duration of an experiment. For example they can pause their subscription. Now events aren't unique.

Say we have two customers in an experiment, but one pauses twice during the experiment. We can't define the the pause rate as total_pauses/total_people as, in this case, the pause rate can be over 100%.

What do we do in this situation?

1. Define rate metrics to be whether a customer did a reccuring action over total number of customers. So in our example we get a pause rate of 50%
2. Switch to count metrics, where we report the number of times an action occurred per customer. So we'd get 1 pause per customer in our example (2 pauses/2 people).

Is there anything else? Would we need to modify our inference procedures in any way (e.g. use a different statistical test, other than a t-test)?

Do we have to modify the test we use to compute p-values when we double count? Or do we modify the things we count to stop double counting?

Answer 1

modify the test we use to compute p-values when we double count?

when we double count the sample, the iid assumption doesn't hold, then the A/B test is more like a cluster-randomized experiment: we randomized allocate treatments to groups, but measure outcomes at the level of the individuals that compose the groups. Many methods like cluster standard error, delta-method, and cluster bootstrap, have been proposed (A/B testing ratio of sums), to give the correct standard error when samples are correlated. What I'm not quite sure about is whether the conclusions still have causal significance in this case. Since each user may have multiple visits, and the previous visit may determine whether the user visits again, this may cause temporal confounding.

switch to count metrics

In this case, Although the distribution of the count metric is far from a normal distribution, in practice, the usual t-test gives acceptable results(Robustness of the Student t-test to non-Gaussian data).

I would recommend the second approach.

Answer 2

First define a metric that monitors the customer behavior you are interested and that is related to the business goal you want to optimize for. Then pick an appropriate statistical procedure.

Take your proposed count metric. What if 10% of customers are "indecisive" and pause two or more times, 10% of customer don't pause at all and the remaining 80% pause once. The count metric gives 1 pause per customer, on average. Is this a useful summary (of the hypothetical situation)? Or perhaps it's better to look at the probability that a customer doesn't pause at all? Statistics cannot tell you what's the best metric for your business/product.