I am trying to determine colors to best grab a web-users attention.
I have a simple system which randomly chooses one of 4 foreground colors, and one of 6 background colors.
I then save the stats in a table like this:
fore_color back_color shown clicked
---------- ---------- ----- -------
red green 16 1
blue green 15 2
At the moment, I require every combination to be shown 250 times before I come to a conclusion about which combination is best... For a test size of ~6000 views.
I would like to know how I can reduce the size of the test while maintaining high confidence that we are making the correct choice (>90% admittedly, I don't know if my current method meets that standard).
Because I am only interested in 90% confidence for the best result, and not at all interested in stats for the losers, we can obviously disqualify early-bad-performers more quickly than final-contenders.
To be clear, my goal is to attain 90% confidence of the best results while showing each combination the least amount of times possible. I will be running this test over 1000 times on many different web pages, and would like to eventually include more variables (like font size).
If the math is simple, I could implement this directly in my own programing. If it's more complex, I'd be more interested in a (preferably free) package that I could feed the data through, and return confidence levels that a particular combination is or is not our winner.
UPDATE: Here is a help page from google website optimizer that does exactly what I am looking for: http://support.google.com/adwords/bin/answer.py?hl=en&answer=61146#results
I would like to be able to reproduce this type of report locally. Using google website optimizer is not a good choice for me because I would need to enter in every combination for >1000 tests by hand.
