1
$\begingroup$

I have 100K customers from a population with a KPI name "spent" which is lognormal distributed.

The customers were split into 2 groups - 1st group - customers with an id ending with 0 - control 2nd group - customers with an id ending with 1-9 - treatment

The treatment group got 20 marketing campaigns during 2022. during 2021 no marketing campaign was made - for both groups.

When comparing the difference between the groups I can see 10% uplift in terms of total spent (after multiplying by 9 to overcome the size difference). The uplift is already exist in the groups even before the campaigns. This uplift is sourced to less then 0.1% of the customers - some of the customers spent a lot.

Does it make sense to compare the total spent in order to evaluate the effectiveness of the marketing campaigns? if not, why?

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ Why would you multiply the observations in one group by 9 just because the other group is nine times as large? What kind of comparison between the two groups did you have in mind? What is the treatment you plan on applying to one group? $\endgroup$
    – Stephan Kolassa
    Commented Jan 30, 2023 at 21:43
  • $\begingroup$ I measure the total spent each group has made. The treatment group gets some marketing campaigns while the control group doesn't. Since I'm measuring a total spent and not average/median it's necessary to overcome to differences in the groups sizes (I can't change the groups allocation/size) $\endgroup$
    – Moti laluom
    Commented Jan 31, 2023 at 7:53
  • 1
    $\begingroup$ Please edit the body of your question to add relevant information, rather than in comments. It's not clear whether you have performed the experiment or plan do to so. And you mention "some marketing campaigns" (plural), not "campaign". How many campaigns are there? $\endgroup$
    – dipetkov
    Commented Jan 31, 2023 at 8:28
  • $\begingroup$ If you only look at total spend, then the distribution is not all that relevant (except insofar as it makes total spend more variable). You are comparing only two data points. If that is what you want to do, then yes, you can scale total spend. Which is actually the precise same thing as calculating and comparing average spend in the two groups, which you would calculate by dividing each group's total spend by the number of individuals in each group. $\endgroup$
    – Stephan Kolassa
    Commented Jan 31, 2023 at 10:04
  • 1
    $\begingroup$ But you won't see that this is due to 0.1% of your customers if all you look at is total spend per group. That is why I believe that looking at total spend is too coarse. Your unit of measurement should be your separate customers. $\endgroup$
    – Stephan Kolassa
    Commented Jan 31, 2023 at 19:35

1 Answer 1

Reset to default
1
$\begingroup$

A few things here:

The customers were split into 2 groups - 1st group - customers with an id ending with 0 - control 2nd group - customers with an id ending with 1-9 - treatment

This is not proper randomization. Probably fine for a one-experiment, but if you use this scheme in repeated experiments, then you will be testing systematic differences between those with an ID ending in 0, and everyone else. Its best to literally randomize using a pRNG or something. Additionally, by only putting users with an ID ending in 0 in the control, you've hurt yourself with respect to power. The most efficient designs are 50/50 designs. Aside from that...

When comparing the difference between the groups I can see 10% uplift in terms of total spent (after multiplying by 9 to overcome the size difference).

I understand why you did this, and I think its a bad idea. The total amount spent is simply the product of the sample mean and the sample size. Since we know a lot about the sample mean, I think its better to analyze that.

The uplift is already exist in the groups even before the campaigns. This uplift is sourced to less then 0.1% of the customers - some of the customers spent a lot.

This is unsurprising to me, many types of comapnies have so called "whales": users who spend thousands of dollars and drive the majority of the revenue.

What I might suggest is that you use log spend instead. This can reign in the spend of whales so that OLS is other regression methods are more tenable. Then, you can use something like a difference in difference as seanv507 notes. However, if there are customers with spend=0 then this won't work for obvious reasons (and doing log 1 + spend doesn't give you valid estimates of the ATE either [1].

Does it make sense to compare the total spent in order to evaluate the effectiveness of the marketing campaigns? if not, why?

As I mentioned above, "No" because we know more about the properties of the mean and the mean is a rescaled total spend.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the helpful comment. I'll explain it a bit better - the method I described was already used in the organization. I understand it's not optimal (or even correct) and I need to explain and provide proof about why it's not optimal/correct (mainly because of the current numbers - an uplift for the treatment group means that the marketing campaigns are working). Once I explained the problems with the method - I'll be able to suggest a better method. $\endgroup$
    – Moti laluom
    Commented Feb 1, 2023 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.