Suppose you conducted an A/B test in which customers were randomized using the hashing technique (Kohavi et al., 2007 etc.). We can thus expect the control and treatment group to have approximately equal number of customers.
In our database, we have a list of tuples (variant_id, customer_number, amount_of_purchase), where variant_id is either control or treatment. Importantly, we only have a record if a customer made a purchase (if they have not, there is no row for them).
Now let's say I want to find the confidence interval for amount_of_purchase that allows us to conclude whether the treatment resulted in better outcome. If I naively calculate it from the records I have, it would yield misleading results because I'm excluding customers who had no purchase.
E.g. if we had:
(treatment, 123, 100 EUR)
(control, 456, 80 EUR)
(control, 789, 80 EUR)
It would be wrong to conclude treatment is better. Now if we had all the records, like this:
(treatment, 001, 0 EUR)
(treatment, 002, 0 EUR)
(treatment, 123, 100 EUR)
(control, 456, 80 EUR)
(control, 789, 80 EUR)
(control, 003, 0 EUR)
Then it would be simple, but let's say we don't have access to records with amount_of_purchase=0, nor do we know how many users were assigned to each control or treatment.
I came up with the following method: we come up with some hash function, which we apply to the customer_number to obtain a bucket (let's say an integer between 0 and 99). We could then calculate the sum of amount_of_purchase for each bucket, using which we can get the confidence intervals as follows:
confidence_lower = sorted_sums[int(0.05 * len(sorted_sums))]
confidence_upper = sorted_sums[int(0.95 * len(sorted_sums))]
We can then compare control and treatment. This idea sounded alright to me but a colleague of mine was not sure if it has problems.
