I'm trying to read between the lines of your question ad to find a practical approach to the problem behind:
1. from methodological point of view you need to be good in design of experiments; a good book (a bible) on this topic is Design and Analysis of Experiments by Douglas Montgomery; Minitab (proprietary software) is very versatile and friendly on this but also R is very good (e.g please see package "agricolae")
2. from technical point of view, for a practical implementation the free tool Google Web Optimizer is a good starting point; it allows you to implement and feed with data the the research; (you can us it complementary with Google Analytics)
This approach is not only for advertising; this two ingredients allows you to plan and implement strategic research via web e.g segmentation of web customers, targeting, pricing analysis etc.
All above can be just a stating and background point; to scale you will need a home brewed methodological approach and technical stuff (e.g research plan, proper experimental designs and analysis; also on software side: specialized web cookies, a MongoDB database or whatever to log data in efficient way, and all of these just to solve your formulated business questions).
NB: IMHO "machine learning" is not the best word in your question and not a feasible answer to the problem.