# Is this method of finding confidence intervals sound (bootstrap like approach)?

Suppose you conducted an A/B test in which customers were randomized using the hashing technique (Kohavi et al., 2007 etc.). We can thus expect the control and treatment group to have approximately equal number of customers.

In our database, we have a list of tuples (variant_id, customer_number, amount_of_purchase), where variant_id is either control or treatment. Importantly, we only have a record if a customer made a purchase (if they have not, there is no row for them).

Now let's say I want to find the confidence interval for amount_of_purchase that allows us to conclude whether the treatment resulted in better outcome. If I naively calculate it from the records I have, it would yield misleading results because I'm excluding customers who had no purchase.

(treatment, 123, 100 EUR)
(control, 456, 80 EUR)
(control, 789, 80 EUR)


It would be wrong to conclude treatment is better. Now if we had all the records, like this:

(treatment, 001, 0 EUR)
(treatment, 002, 0 EUR)
(treatment, 123, 100 EUR)
(control, 456, 80 EUR)
(control, 789, 80 EUR)
(control, 003, 0 EUR)


Then it would be simple, but let's say we don't have access to records with amount_of_purchase=0, nor do we know how many users were assigned to each control or treatment.

I came up with the following method: we come up with some hash function, which we apply to the customer_number to obtain a bucket (let's say an integer between 0 and 99). We could then calculate the sum of amount_of_purchase for each bucket, using which we can get the confidence intervals as follows:

confidence_lower = sorted_sums[int(0.05 * len(sorted_sums))]
confidence_upper = sorted_sums[int(0.95 * len(sorted_sums))]


We can then compare control and treatment. This idea sounded alright to me but a colleague of mine was not sure if it has problems.

• Your objective is unclear because you haven't defined the property you are trying to estimate. If a person does not make a purchase, they don't even qualify as a "customer" in the usual sense of the word. If you do want to include some set of non-purchasers, then you have no hope of estimating (say) the average purchase per person in the population unless you have some information about the rest of the population. A hash function, by its very construction, does not supply such information.
– whuber
Commented Jun 23, 2023 at 17:59
• @whuber added edit, hope this clarifies. The property being estimated is the total sum divided by len(bucket), and it is necessary to include customers who did not purchase anything during the experimentation period because the treatment is expected to work by enticing customers to make a purchase (imagine amazon etc. where people have accounts and sometimes make purchases). Finally the information I have on the rest of the population is: I know both control and treatment has the same population size, though I don't know how big. Commented Jun 24, 2023 at 9:50