My website has advertising for various companies - say 10,000 companies. Each company has it's own page but all the pages are currently similar except by which company is on the page. A success/response/dependent variable is a visitor clicking on the advertisement. Each company's page has a different success rate - whereby 50% of visitors click for some company pages, and 1% for others.
I am testing a new page layout that will be for every company. We have 4 different potential new pages, for a total of 5 pages being tested including the control. A randomization algorithm will assign each visitor into one of the 5 buckets, however this does not guarantee that each bucket will get an equal representation of stores. Whereby, one bucket could get more visitors to stores with the 50% success rate, or vice versa.
How big of an issue is this?
A possible solution is to serially (not randomly) assign visitors into buckets based on the store. So for Store A, visitor 1 to that store gets in bucket 1, visitor 2 bucket 2, and all the way to 5, whereby visitor 6 goes to bucket 1 again. Is this necessary or even statistically valid or does it create bias?
Thanks - i am happy to explain more or even to summarize this in a more universal format if that is preferred.