... (optional) within the context of Google Web Optimizer.
Suppose you have two groups and a binary response variable. Now you get the following outcome:
- Original: 401 trials, 125 successful trials
- Combination16: 441 trials, 141 successful trials
The difference is not statistically significant, however one can calculate a probability that Combination16 will beat Original.
To calculate "Chance to beat Original" I have used an bayesian approach, i.e. performing a two-dimensional monte carlo integration over the bayesian-style confidence intervals (beta-distribution, (0,0) prior). Here is the code:
trials <- 10000 resDat<-data.frame("orig"=rbeta(trials,125+1,401-125+1), "opt"=rbeta(trials,144+1,441-144+1)) length(which(resDat$opt>resDat$orig))/trials
This results in 0.6764.
Which technique would a frequentist use to calculate "Chance to beat ..." ? Maybe the power function of Fisher's exact test ?
Optional: Context of Google Web Optimizer
Google Web Optimizer is a tool for controlling multivariate Testing or A/B-Testing. This only as an introduction since this should not matter for the question itself.
The example presented above was taken from the explanation page of Google Web Optimizer (GWO), which you can find here (please scroll down to the section "Estimated Conversion Rate Ranges"), specifically from figure 2.
Here GWO delivers 67.8% for "Chance to beat Original", which slightly differs from my result. I guess Google uses a more frequentist-like approach and I wondered: What could it be ?
EDIT: Since this question was close to disappear (I guess because of its too specific nature), I have rephrased it to be of general interest.