1
$\begingroup$

Background: Our business wishes to perform an demand analysis based on two marketing treatments A and B. Due to constraints imposed from the onset, there will be a 50/50 split (receive/not-receive) for treatment A, but a 90/10 split for treatment B. This results in a 2x2 treatment matrix as such:

      B    noB
  A   45%  5%
noA   45%  5%

We used a randomizing method to assign customers to each group, following the distribution outlined above. Before commencing our test, we checked a basic parameter (email addr. per cust) and found that within the B/noB group there is a significant (by GLM) difference. No sig difference is found between quadrants or between A/noA. Reminder, this was just a check to see if we are starting with equivalent groups.

Question: Because we already know that there is variation in our groups, and assuming we are bound to the unequal distribution of customers to treatment group, are we doomed in terms of statistical analysis after we run our test? Are there precautions we can take before running the test? Should we re-sample our groups using another method? Any advice would be greatly appreciated, even if it is pointing to another thread or academic article.

Please let me know if this is unclear...

Thanks!

$\endgroup$
  • $\begingroup$ How confident are you that this parameter has an impact on the outcome of the test ? If I understand it right, you have not run it yet, right ? $\endgroup$ – brumar May 26 '15 at 20:11
  • $\begingroup$ Pretty confident based on historical data. One thing I didn't make clear earlier, is that multiple custs can share an email, but the constraints of the test prevent two customers who share an email from being in two different treatment groups. Customers can also have multiple email addresses that are share (or not) with others. This means that when we randomize into groups, we have to also pull in the 'network' of customers created by email sharing chains. It is this issue that introduces variation into our samples. The error bleeds into other variables such as historical performance. $\endgroup$ – normal_guy May 27 '15 at 19:36
1
$\begingroup$

Are we doomed in terms of statistical analysis after we run our test?

I don't think so, but if you can't balance it, you should definitely consider this factor in your analysis (at the cost of some statistical power I think)

What interrogates me more is this network of customer you also mentioned in your comment. If they know each other, they could react the same way, don't they ? It implies that the random variable you are about to inspect would not be strictly i.i.d, even more if you study an effect around communication (just a guess relatively to your design). I don't know if it's a big problem or how to address it in a general way.

$\endgroup$
0
$\begingroup$

You could do some sort of adaptive randomization. Since you already know the number of email addresses per customer you could use minimization to control the number of customers in a particular group.

For example, if there are a lot of people in group B with multiple email addresses, minimization would allocate the rest of the sample into the no B group.

$\endgroup$
  • $\begingroup$ Thanks for the link and advice. I'll have to read up on this. $\endgroup$ – normal_guy May 27 '15 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.