I am a software developer working on A/B testing systems. I don't have a solid stats background but have been picking up knowledge over the past few months.
A typical test scenario involves comparing two URLs on a website. A visitor visits
LANDING_URL and then is randomly forwarded to either
URL_EXPERIMENTAL. A visitor constitutes a sample, and a victory condition is achieved when the visitor performs some desirable action on that site. This constitutes a conversion and the rate of conversion rates is the conversion rate (typically expressed as a percentage). A typical conversion rate for a given URL is something in the realm of 0.01% to 0.08%. We run tests to determine how new URLs compare against old URLs. If
URL_EXPERIMENTAL is shown to outperform
URL_CONTROL, we replace
We have developed a system using simple hypothesis testing techniques. I used the answers to another CrossValidated question here to develop this system.
A test is set up as follows:
- The conversion rate estimate
URL_CONTROLis calculated using historical data.
- The desired target conversion rate
- A significance level of 0.95 is typically used.
- A power of 0.8 is typically used.
Together, all of these values are used to compute the desired sample size. I'm using the R function
power.prop.test to obtain this sample size.
A test will run until all samples are collected. At this point, the confidence intervals for
CR_EXPERIMENTAL are computed. If they do not overlap, then a winner can be declared with significance level of 0.95 and power of 0.8.
The users of our tests have two major concerns, though:
1. If, at some point during the test, enough samples are collected to show a clear winner, can't the test be stopped?
2. If no winner is declared at the end of the test, can we run the test longer to see if we can collect enough samples to find a winner?
It should be noted that many commercial tools out there exist that allow their users to do exactly what our own users desire. I've read that there are many fallacies with the above, but I've also come across the idea of a stopping rule and would like to explore the possibility of using such a rule in our own systems.
Here are two approaches we would like to consider:
power.prop.test, compare the current measured conversion rates to the current number of samples and see if enough samples have been collected to declare a winner.
Example: A test has been set up to see if the following behavior exists in our system:
CRE_EXPERIMENTAL: 0.1 * 1.3
- With these parameters, the sample size
However, as the test advances and reaches 325 samples,
CRM_CONTROL (measured conversion rate for control) is 0.08 and
CRM_EXPERIMENTAL is 0.15.
power.prop.test is run on these conversion rates and
N is found to be 325. Exactly the number of samples needed to declare
CRM_EXPERIMENTAL to be the winner! At this point it is our hope that the test could be ended. Similarly, if the test reaches 1774 samples but no winner is found, but then it reaches 2122 samples which is enough to show that
CRM_CONTROL of 0.1 and
CRM_EXPERIMENTAL 0.128 is a result where a winner can be declared.
In a related question users advised that such a test is less credible due to encouraging early stops having fewer samples and also being vulnerable to estimation bias and an increased number of Type I and Type II errors. Is there some way to make this stopping rule work? This is our preferred approach since it means less programming time for us. Perhaps this stopping rule could work by offering some kind of numerical score or scores that measures the credibility of the test should it be stopped early?
These methods of testing are designed exactly for the situation we find ourselves in: how can our users start a test and end it in such a way that they don't waste excess time in testing? Either running a test too long, or having to start a test over with different parameters.
Of the two above methods, I favor SPRT because the mathematics is a bit easier for me to grasp and because it looks like it may be easier to program. However, I don't understand how to use the likelihood function in this context. If someone could construct an example of how to compute the likelihood-ratio, the cumulative sum of the likelihood-ratio, and continue through an example illustrating a situation when one would continue monitoring, when one would accept the null hypothesis and the alternative hypothesis, that would help us determine if SPRT is the right way to go.