Background: Our business wishes to perform an demand analysis based on two marketing treatments A and B. Due to constraints imposed from the onset, there will be a 50/50 split (receive/not-receive) for treatment A, but a 90/10 split for treatment B. This results in a 2x2 treatment matrix as such:
B noB A 45% 5% noA 45% 5%
We used a randomizing method to assign customers to each group, following the distribution outlined above. Before commencing our test, we checked a basic parameter (email addr. per cust) and found that within the B/noB group there is a significant (by GLM) difference. No sig difference is found between quadrants or between A/noA. Reminder, this was just a check to see if we are starting with equivalent groups.
Question: Because we already know that there is variation in our groups, and assuming we are bound to the unequal distribution of customers to treatment group, are we doomed in terms of statistical analysis after we run our test? Are there precautions we can take before running the test? Should we re-sample our groups using another method? Any advice would be greatly appreciated, even if it is pointing to another thread or academic article.
Please let me know if this is unclear...