# How to deal with unequal proportions in an A/B test?

I have 100 customers, 40 are Females and 60 are Males. My marketing team has created 2 separate campaigns with different offers for both groups. We create an A/B test for each group to study campaign lift. Below are the splits (A = No offer | B = offer depending on the group)

Counts :

    Group  |  A  |  B  | Total
-----------------------------
Females |  10 |  30 |  40
Males   |  20 |  40 |  60
-----------------------------
Total   |  30 |  70 | 100


Here are the results,

 Results : $value purchased per person during active campaign dates Again, A = Control, B = Test... so lift over control = (B-A)/A Group | A | B | Lift ----------------------------- Females |$5 |  $7 | 40% Males |$3 |  $4 | 33% ----------------------------- Wt. Avg |$3.6| \$5.3|  47%


What's funny is the lift of Total-A vs. Total-B is more than individual groups, and I realise that this has got something to do with the proportion of A:B across groups. (Females = 1:3, Males = 1:2)

My question is what the best way to solve for this discrepancy ?

The problem arises from the weighting, not just of the genders (which I understand is representative of your customers) but it is an artefact of the A/B split, which is different between the genders. If you re-do the calculation, but both males and females have equal numbers in each condition, you have the following.
Uplift for females 40% (as you calculated)
Uplift for males 33% (as you calculated)
Unweighted mean uplift (equal numbers of males and females) 38%
Weighted mean uplift (males:females 3:2) 37%; between the two figures, but closer to the male figure because there are more males.
Weighted mean uplift (males:females 2:1) 36%; still between the two figures, but even closer to the male figure because there are even more males.

See screenshots below; let me know if you need the formulas.

• Thanks for your response Mike. To clarify why I took weighted average, because the underlying sales have been different for different groups (A&B) within Females and Males, without a weighted average how can I attribute good/poor campaign sales for Total-As and Total-Bs. Thanks for the intro to Simpson's paradox, and you're right frequency data aggregation can cause this often. Unfortunately, management does not care. :(
– user125518
Aug 3 '16 at 15:40
• OK. I think to understand the situation properly, I would need more information. Would you normally expect there to be this proportion of males to females in your customer base? And why were 3/4 of female customers given the offer, but only 2/3 of male customers? Aug 4 '16 at 15:54
• To answer your questions, 1. Yes F:M proportion is similar to population. 2. Regarding the unequal proportion in the offer sample, Business decided that giving the offer to 25% of females and 33% of the males is a "safe-bet".. essentially hedging risk of the offer not working vs. enough sample size for measuring a lift with certain confidence. Ofcourse, there are more than 100 customers..this is just for illustration. Thanks again. Aug 4 '16 at 22:26
• OK, I think I have it now, see amended answer. Aug 8 '16 at 16:19