Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am looking for theoretical resources (books, tutorials, etc.) to learn about making sound statistical inferences given (plenty of) multivariate website conversion data.

I'm after the math involved, and cannot find any good non-marketing stuff on the web. The sort of questions I want to answer: how much impact does a single variable (e.g. color of text) have? what is the correlation between variables? what type of distribution is used for modelling (Gaussian, Binomial, etc.)? When using statistics to analyze results - what should be considered as a random variable - the web-page element that gets different variations or the binary conversion-or-no-conversion outcome of an impression?

There's plenty of information about different website optimization testing methods and their benefits\pitfalls, plenty of information about multivariate statistics in general, do you guys know of resources that discuss technical statistics in this specific context of website optimization ?

Thanks for any info!

share|improve this question
I am dealing with roughly similar issues in my daily work live. Even pushing simple ab-testing beyond those easy-cheesy binary metrices and make it usable in a commercial environment was only possible by first studying statistical methods independent of the application area and THEN, now armed with the correct mindset, apply this methods to the web. To the best of my knowledge, there is no such book, but I guess it would be a bestseller. Good luck ! – steffen Dec 2 '11 at 9:04
up vote 5 down vote accepted

This Microsoft page has quite a few resources.

I suggest you read at least this paper from the page: "Controlled Experiments on the Web: Survey and Practical Guide." It'll give you some starting point about the metrics to measure and convey, things to consider for online experiments (including designing elements of web pages), and the related statistics.

Enjoy! -Al

share|improve this answer
(+1) good resources. Especially Kohavi did a great job writing about the optimization he performed at amazon. – steffen Dec 2 '11 at 9:08
Great stuff! Thanks a lot, definitely a valuable resource. – bloodcell Dec 4 '11 at 10:14

These lectures notes are more about optimizing online advertising than optimizing a website, but the references there (especially lecture 6) might put you in the right direction.

I hope it helps.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.