5
$\begingroup$

My website has advertising for various companies - say 10,000 companies. Each company has it's own page but all the pages are currently similar except by which company is on the page. A success/response/dependent variable is a visitor clicking on the advertisement. Each company's page has a different success rate - whereby 50% of visitors click for some company pages, and 1% for others.

I am testing a new page layout that will be for every company. We have 4 different potential new pages, for a total of 5 pages being tested including the control. A randomization algorithm will assign each visitor into one of the 5 buckets, however this does not guarantee that each bucket will get an equal representation of stores. Whereby, one bucket could get more visitors to stores with the 50% success rate, or vice versa.

How big of an issue is this?

A possible solution is to serially (not randomly) assign visitors into buckets based on the store. So for Store A, visitor 1 to that store gets in bucket 1, visitor 2 bucket 2, and all the way to 5, whereby visitor 6 goes to bucket 1 again. Is this necessary or even statistically valid or does it create bias?

Thanks - i am happy to explain more or even to summarize this in a more universal format if that is preferred.

$\endgroup$
  • $\begingroup$ We are grouping our stores into blocks/bins by their click-through rates (CTR). That way we can look at the distribution of stores based on a "small" amount of BINS instead of 10,000 stores. This way it will be easier to compare distribution. So thinking of the data like a histogram, the BINS on the X axis will be groups of stores with a similar CTR. The Y axis will be counts of visitors in those BINS. $\endgroup$ – mcholt Oct 4 '11 at 17:26
3
$\begingroup$

Your problem is similar to clinical randomized trials whereby one wants to stratify on a certain clinical condition. For example, in trials of cardiac surgeries, one may want to ensure that an equal number of people with congestive heart failure end up in each of the groups (intervention and control).

So, a separate randomization list is made up for people with congestive heart failure and those without, ensuring (over time) that equal numbers of people with congestive heart failure end up in each group.

In your case, for this approach, you would need 10,000 randomization lists: one for each company. This is possible to do with automation. Within each randomization list, you would block by groups of five so that, each five hits to that particular company would exhaust the five possible page types, but their order within each group of five would be randomly assigned.

An alternative is just to stick to simple randomization. If you have a very large number of hits, you will still get reasonable balance.

$\endgroup$
  • $\begingroup$ "An alternative is just to stick to simple randomization. If you have a very large number of hits, you will still get reasonable balance." - what is a very large number? my calculation of sample size, ignoring the 10,000 pages problem, gives me a sample size of 50,000 needed for each bucket, if I doubled this (100,000) would that be large enough? $\endgroup$ – mcholt Sep 30 '11 at 19:35
  • 2
    $\begingroup$ The idea of stratified blocked randomization is sound, but having too many strata defeats the point of stratified randomization. Assuming you have some idea which companies have high/low click-through rates, you could group together companies with similar rates, and have those groups form 5-10 strata. $\endgroup$ – Aniko Sep 30 '11 at 20:24
  • $\begingroup$ Aniko, we are grouping our stores into blocks/bins by their click-through rates. That way we can look at the distribution of stores based on X amount of Bins instead of 10,000 stores. $\endgroup$ – mcholt Oct 4 '11 at 17:22
0
$\begingroup$

If your randomization process is exogenous to the relationships you are measuring, your results should be unbiased.

If you are interested in click-through rates of the different treatments, you will be considering at the differences in the average rate. The number of items in each bucket will influence your confidence in the average rate you come up with and, by extension, your confidence that this value is different from another treatment. But the number of items in the bucket won't bias the average.

In cases where you want to compare the distribution of some qualities between two populations to see if the distributions are different, you can't select on those qualities because the size of the buckets is what you are investigating. It's critical to not select on your dependent variable. But it doesn't sounds like you're doing that here.

In fact, if your randomization process is any good, you probably won't have exactly the same number of items in each bucket. And that's fine. :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.