I have 100 customers, 40 are Females and 60 are Males. My marketing team has created 2 separate campaigns with different offers for both groups. We create an A/B test for each group to study campaign lift. Below are the splits (A = No offer | B = offer depending on the group)

Counts :

    Group  |  A  |  B  | Total
   Females |  10 |  30 |  40
   Males   |  20 |  40 |  60
   Total   |  30 |  70 | 100

Here are the results,

 Results : $ value purchased per person during active campaign dates 
 Again, A = Control, B = Test... so lift over control = (B-A)/A 

    Group  |  A  |  B  | Lift
   Females |  $5 |  $7 |  40%
   Males   |  $3 |  $4 |  33%
   Wt. Avg | $3.6| $5.3|  47%

What's funny is the lift of Total-A vs. Total-B is more than individual groups, and I realise that this has got something to do with the proportion of A:B across groups. (Females = 1:3, Males = 1:2)

My question is what the best way to solve for this discrepancy ?


(Amended answer following discussion)
The problem arises from the weighting, not just of the genders (which I understand is representative of your customers) but it is an artefact of the A/B split, which is different between the genders. If you re-do the calculation, but both males and females have equal numbers in each condition, you have the following.
Uplift for females 40% (as you calculated)
Uplift for males 33% (as you calculated)
Unweighted mean uplift (equal numbers of males and females) 38%
Weighted mean uplift (males:females 3:2) 37%; between the two figures, but closer to the male figure because there are more males.
Weighted mean uplift (males:females 2:1) 36%; still between the two figures, but even closer to the male figure because there are even more males.

See screenshots below; let me know if you need the formulas.

Screenshot of spreadsheet

  • $\begingroup$ Thanks for your response Mike. To clarify why I took weighted average, because the underlying sales have been different for different groups (A&B) within Females and Males, without a weighted average how can I attribute good/poor campaign sales for Total-As and Total-Bs. Thanks for the intro to Simpson's paradox, and you're right frequency data aggregation can cause this often. Unfortunately, management does not care. :( $\endgroup$
    – user125518
    Aug 3 '16 at 15:40
  • $\begingroup$ OK. I think to understand the situation properly, I would need more information. Would you normally expect there to be this proportion of males to females in your customer base? And why were 3/4 of female customers given the offer, but only 2/3 of male customers? $\endgroup$
    – MikeG
    Aug 4 '16 at 15:54
  • $\begingroup$ To answer your questions, 1. Yes F:M proportion is similar to population. 2. Regarding the unequal proportion in the offer sample, Business decided that giving the offer to 25% of females and 33% of the males is a "safe-bet".. essentially hedging risk of the offer not working vs. enough sample size for measuring a lift with certain confidence. Ofcourse, there are more than 100 customers..this is just for illustration. Thanks again. $\endgroup$ Aug 4 '16 at 22:26
  • $\begingroup$ OK, I think I have it now, see amended answer. $\endgroup$
    – MikeG
    Aug 8 '16 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.