# Math behind multivariate testing for website optimization

I am looking for theoretical resources (books, tutorials, etc.) to learn about making sound statistical inferences given (plenty of) multivariate website conversion data.

I'm after the math involved, and cannot find any good non-marketing stuff on the web. The sort of questions I want to answer: how much impact does a single variable (e.g. color of text) have? what is the correlation between variables? what type of distribution is used for modelling (Gaussian, Binomial, etc.)? When using statistics to analyze results - what should be considered as a random variable - the web-page element that gets different variations or the binary conversion-or-no-conversion outcome of an impression?

There's plenty of information about different website optimization testing methods and their benefits\pitfalls, plenty of information about multivariate statistics in general, do you guys know of resources that discuss technical statistics in this specific context of website optimization ?

Thanks for any info!

-
I am dealing with roughly similar issues in my daily work live. Even pushing simple ab-testing beyond those easy-cheesy binary metrices and make it usable in a commercial environment was only possible by first studying statistical methods independent of the application area and THEN, now armed with the correct mindset, apply this methods to the web. To the best of my knowledge, there is no such book, but I guess it would be a bestseller. Good luck ! – steffen Dec 2 '11 at 9:04

This Microsoft page has quite a few resources.

I suggest you read at least this paper from the page: "Controlled Experiments on the Web: Survey and Practical Guide." It'll give you some starting point about the metrics to measure and convey, things to consider for online experiments (including designing elements of web pages), and the related statistics.

Enjoy! -Al

-
(+1) good resources. Especially Kohavi did a great job writing about the optimization he performed at amazon. – steffen Dec 2 '11 at 9:08
Great stuff! Thanks a lot, definitely a valuable resource. – bloodcell Dec 4 '11 at 10:14

These lectures notes are more about optimizing online advertising than optimizing a website, but the references there (especially lecture 6) might put you in the right direction.

http://www.stanford.edu/class/msande239/

I hope it helps.

-