I have 5 treatments that I'm going to randomly assign to visitors to my website. I'll have 10,000 or more visitors. Is it statistically valid to analyze the results based on visitor characteristics afterwards and in multiple ways? For instance, can I validly do an analysis where I pretend I had assigned visitors to blocks based on gender and age before I randomly assigned a treatment to them? Is it then valid to do another analysis on the same visitors where I pretend I had randomly assigned them to blocks based on height and weight before I randomly assigned a treatment? I know I'll lose some power because my sample sizes won't be exactly the same but, provided each block has a large number of visitors and each of the 5 treatments are well represented in each block will this work? For all The treatments, the effect is pass/fail. What would be the best statistical test to run to compare the treatments?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.