We are running a marketing campaign where we send out comminications to a select group of customers on daily basis. The customers are different each day.
However each day we only have a small number of total customers, average 9 with SD 7.

Each day we randomise the sample into control(10%) and treatment. We round the control group to the nearest integer. For example, if we have 5 customers we allocate 1 to control.
The aim is to look at the behaviour of customers 30 days after email.

To get a sufficient sample size we aggregate samples of many days, so after 10 months we have around 200 control and 1700 treatment.

My question is whether there is any bias or a problem of allocating samples to a control group from such small sample sizes and then adding them all together to do z-test of proportions?

  • $\begingroup$ When are you carrying out the test? Every day? just once? How confident are you that the proportions will not change over time? $\endgroup$ – Glen_b Aug 7 '18 at 3:00
  • $\begingroup$ I'm doing the actual test after 10 months, this is when we have sufficient sample size to do the test, the emails get sent out daily and we split control/treatment daily. It's just one test to check whether the proportion of customers in Control had different behaviour to customers in Treatment. I want to know whether the process of selecting control/treatment groups in this way is appropriate. One issue that we get is that if there are only 5 customers in the sample, we will allocate 1 to control. But if there are 20 customers, we will alocate 2 to control (10%). $\endgroup$ – jmich738 Aug 7 '18 at 3:42
  • $\begingroup$ Then I take it it's not unthinkable some customers are in this data set more than once? $\endgroup$ – Frans Rodenburg Aug 7 '18 at 4:38
  • $\begingroup$ they are, but not because of sampling. The nature of the behaviour we are tracking means that customers can appear at most 2-3 times over 12 month period and if they are in the same again, that's because they should be there. I don't think that customers appearing multiple times is really an issue for us. It's more the validity of the control/treatment allocation process. I tried to look this up online, but I can't find the term for this process. $\endgroup$ – jmich738 Aug 7 '18 at 4:52
  • $\begingroup$ I think that customers appearing more than once really is an issue for you $\endgroup$ – Glen_b Aug 7 '18 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.