I'm trying to analyze the results of an A/B test, specifically around losses. We had two groups, who had bought something, and then were put into our A/B test to minimize potential losses from their purchases. So our treatment did not affect their purchase volume, but potentially the subsequent losses.

Say we have two groups: A: losses = $500, purchases = $1500 B: losses = $520, purchases = $1550

The loss amount is partly dependent on our treatment (we did something to try to reduce losses), but also on the purchases (if people buy more, they are more likely to have losses, just because of greater volume).

So if I do a t-test on the losses, I'm ignoring the effect of the purchase amounts. However, if I do a proportions test, I am 'fooling' the test into thinking I had (500, 520) successes and (1500, 1550) trials, which is not really the case.

What's the right approach here? Thank you!!

  • $\begingroup$ What's a loss? If individual $i$ makes a purchase, is there some probability that the purchase is a loss? Is an independent observation essentially a purchase? $\endgroup$ Oct 24 '16 at 10:36
  • $\begingroup$ Each observation is a purchase yes. Say if Buyer A makes one purchase of $100. That would count as one observation, with a value of $100. Then there is a possibility, if what was bought is faulty, that this purchase will turn into a loss (which could be of $100 or less in the case of a partial refund). It's a loss because we don't get the item back. Does this help? $\endgroup$ Oct 24 '16 at 10:40

One possibility is to test the difference in losses as a proportion of sales.

I would do this for each person, before combining into groups. So, if you have 100 people in each group, then you would calculate 200 proportions of losses as sales, then do a t-test on the proportions.

Another is to do a regression with loss as the dependent variable and group and purchase as the independent variables.

However, if people were not randomly assigned to groups, then you'd have to account for that (neither of the above would be correct).

  • 1
    $\begingroup$ Thanks so much Peter. That makes sense to me. The groups were randomly assigned, so no issue there. Really appreciate you taking your time to answer me. $\endgroup$ Oct 24 '16 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.