# Is having cumulative samples a problem?

We are running a marketing campaign where we send out comminications to a select group of customers on daily basis. The customers are different each day.
However each day we only have a small number of total customers, average 9 with SD 7.

Each day we randomise the sample into control(10%) and treatment. We round the control group to the nearest integer. For example, if we have 5 customers we allocate 1 to control.
The aim is to look at the behaviour of customers 30 days after email.

To get a sufficient sample size we aggregate samples of many days, so after 10 months we have around 200 control and 1700 treatment.

My question is whether there is any bias or a problem of allocating samples to a control group from such small sample sizes and then adding them all together to do z-test of proportions?

• When are you carrying out the test? Every day? just once? How confident are you that the proportions will not change over time? – Glen_b Aug 7 '18 at 3:00
• I'm doing the actual test after 10 months, this is when we have sufficient sample size to do the test, the emails get sent out daily and we split control/treatment daily. It's just one test to check whether the proportion of customers in Control had different behaviour to customers in Treatment. I want to know whether the process of selecting control/treatment groups in this way is appropriate. One issue that we get is that if there are only 5 customers in the sample, we will allocate 1 to control. But if there are 20 customers, we will alocate 2 to control (10%). – jmich738 Aug 7 '18 at 3:42
• Then I take it it's not unthinkable some customers are in this data set more than once? – Frans Rodenburg Aug 7 '18 at 4:38
• they are, but not because of sampling. The nature of the behaviour we are tracking means that customers can appear at most 2-3 times over 12 month period and if they are in the same again, that's because they should be there. I don't think that customers appearing multiple times is really an issue for us. It's more the validity of the control/treatment allocation process. I tried to look this up online, but I can't find the term for this process. – jmich738 Aug 7 '18 at 4:52
• I think that customers appearing more than once really is an issue for you – Glen_b Aug 7 '18 at 10:08