18
$\begingroup$

I am a software developer working on A/B testing systems. I don't have a solid stats background but have been picking up knowledge over the past few months.

A typical test scenario involves comparing two URLs on a website. A visitor visits LANDING_URL and then is randomly forwarded to either URL_CONTROL or URL_EXPERIMENTAL. A visitor constitutes a sample, and a victory condition is achieved when the visitor performs some desirable action on that site. This constitutes a conversion and the rate of conversion rates is the conversion rate (typically expressed as a percentage). A typical conversion rate for a given URL is something in the realm of 0.01% to 0.08%. We run tests to determine how new URLs compare against old URLs. If URL_EXPERIMENTAL is shown to outperform URL_CONTROL, we replace URL_CONTROL with URL_EXPERIMENTAL.

We have developed a system using simple hypothesis testing techniques. I used the answers to another CrossValidated question here to develop this system.

A test is set up as follows:

  • The conversion rate estimate CRE_CONTROL of URL_CONTROL is calculated using historical data.
  • The desired target conversion rate CRE_EXPERIMENTAL of URL_EXPERIMENTAL is set.
  • A significance level of 0.95 is typically used.
  • A power of 0.8 is typically used.

Together, all of these values are used to compute the desired sample size. I'm using the R function power.prop.test to obtain this sample size.

A test will run until all samples are collected. At this point, the confidence intervals for CR_CONTROL and CR_EXPERIMENTAL are computed. If they do not overlap, then a winner can be declared with significance level of 0.95 and power of 0.8.

The users of our tests have two major concerns, though:

1. If, at some point during the test, enough samples are collected to show a clear winner, can't the test be stopped?

2. If no winner is declared at the end of the test, can we run the test longer to see if we can collect enough samples to find a winner?

It should be noted that many commercial tools out there exist that allow their users to do exactly what our own users desire. I've read that there are many fallacies with the above, but I've also come across the idea of a stopping rule and would like to explore the possibility of using such a rule in our own systems.

Here are two approaches we would like to consider:

1. Using power.prop.test, compare the current measured conversion rates to the current number of samples and see if enough samples have been collected to declare a winner.

Example: A test has been set up to see if the following behavior exists in our system:

  • CRE_CONTROL: 0.1
  • CRE_EXPERIMENTAL: 0.1 * 1.3
  • With these parameters, the sample size N is 1774.

However, as the test advances and reaches 325 samples, CRM_CONTROL (measured conversion rate for control) is 0.08 and CRM_EXPERIMENTAL is 0.15. power.prop.test is run on these conversion rates and N is found to be 325. Exactly the number of samples needed to declare CRM_EXPERIMENTAL to be the winner! At this point it is our hope that the test could be ended. Similarly, if the test reaches 1774 samples but no winner is found, but then it reaches 2122 samples which is enough to show that CRM_CONTROL of 0.1 and CRM_EXPERIMENTAL 0.128 is a result where a winner can be declared.

In a related question users advised that such a test is less credible due to encouraging early stops having fewer samples and also being vulnerable to estimation bias and an increased number of Type I and Type II errors. Is there some way to make this stopping rule work? This is our preferred approach since it means less programming time for us. Perhaps this stopping rule could work by offering some kind of numerical score or scores that measures the credibility of the test should it be stopped early?

2. Using sequential analysis or SPRT.

These methods of testing are designed exactly for the situation we find ourselves in: how can our users start a test and end it in such a way that they don't waste excess time in testing? Either running a test too long, or having to start a test over with different parameters.

Of the two above methods, I favor SPRT because the mathematics is a bit easier for me to grasp and because it looks like it may be easier to program. However, I don't understand how to use the likelihood function in this context. If someone could construct an example of how to compute the likelihood-ratio, the cumulative sum of the likelihood-ratio, and continue through an example illustrating a situation when one would continue monitoring, when one would accept the null hypothesis and the alternative hypothesis, that would help us determine if SPRT is the right way to go.

$\endgroup$
  • 2
    $\begingroup$ I commend your not resorting to voodoo. When you are using a tool and you do not understand exactly what it is doing or how it works, then you are not qualified to interpret the results of the tool. When an analysis drives a business decision, and you are putting time and money into the result, it shows ownership if you take the time to understand the source of data. It is the kind of hard work that gives you more opportunity instead of being "in the herd". $\endgroup$ – EngrStudent Jul 12 '13 at 13:05
7
+50
$\begingroup$

This is an interesting problem and the associated techniques are have lots of applications. They are often called "interim monitoring" strategies or "sequential experimental design" (the wikipedia article, which you linked to, is unfortunately a little sparse), but there are several ways to go about this. I think @user27564 is mistaken in saying that these analyses must necessarily be Bayesian--there are certainly frequentist approaches for interim monitoring too.

Your first approach resembles one of the original approaches to interim monitoring, called 'curtailment.' The idea is very simple: you should stop collecting data once the experiment's outcome is inevitable. Suppose you've got a collection of 100 $As$ and/or $Bs$ and you want to know whether it was generated by a process that selects an $A$ or $B$ at random each time (i.e., $P(A)=P(B)=0.5$. In this case, you should stop as soon as you count at least 58 items of the same kind; counting the remaining items won't change the significance after that point. The number $58$ comes from finding $x \textrm{ such that } 1-F(x;100;0.5) \lt \alpha$, where $F$ is the cumulative binomial distribution.

Similar logic lets you find the "inevitability points" for other tests where:

  1. The total sample size* is fixed, and
  2. Each observation contributes a bounded amount to the sample.

This would probably be easy for you to implement--calculate the stopping criteria offline and then just plug it into your site's code--but you can often do even better if you're willing to terminate the experiment not only when the outcome is inevitable, but when it is also very unlikely to change.

This is called stochastic curtailment. For example, suppose, in the example above, that we've seen 57 $A$s and 2 $B$s. We might feel reasonably confident, if not absolutely certain, that there is at least one more $A$ in the box of 100, and so we could stop. This review by Christopher Jennison and Bruce Turnbull, works through Stochastic Curtailment in Section 4. They also have a longer book; you can peek at Chapter 10 via Google Books. In addition to the derivation, the book has some formulae where you can more or less plug in the results of your interim tests.

There are a number of other approaches too. Group sequential methods are designed for situations where you may not be able to obtain a set number of subjects and the subjects trickle in at variable rates. Depending on your site's traffic, you might or might not want to look into this.

There are a fair number of R packages floating around CRAN, if that's what you're using for your analysis. A good place to start might actually be the Clinical Trials Task View, since a lot of this work came out of that field.


[*] Just some friendly advice: be careful when looking at significance values calculated from very large numbers of data points. As you collect more and more data, you will eventually find a significant result, but the effect might be trivially small. For instance, if you asked the whole planet whether they prefer A or B, it's very unlikely that you would see an exact 50:50 split, but it's probably not worth retooling your product if the split is 50.001:49.999. Keep checking the effect size (i.e., difference in conversion rates) too!

$\endgroup$
  • 1
    $\begingroup$ To explicitly address your users' concerns: Yes, you can definitely end the analysis early. This happens all the time for clinical trials--the drug is either such a smashing success that they have sufficient data for its efficacy and want to give it to the folks stuck in the control group (or, more likely, the drug is a huge bust/making things worse). However, extending the experiment is more contentious--there are methods for correcting for multiple "looks" but you'd be better off fixing a maximum N in advance--you can always stop early! $\endgroup$ – Matt Krause Jul 10 '13 at 18:38
  • 2
    $\begingroup$ Thanks for this, I totally agree with the stoping if the experiment is inevitable, that really makes sense! With that 'stopping if its really unlikely' I would doubt a real frequentist would agree. It's nothing less then saying: Hey, 95%? I would say 93% is also good! I mean it would be also good to just be satisfied with 90% confidence, but as a frequentist before looking to the data! $\endgroup$ – SebastianNeubauer Jul 10 '13 at 20:08
  • 1
    $\begingroup$ I'm not an arch-frequentist, nor do I play one on TV, but I think you can still have a reasonable frequentist interpretation of an early stopping rule--if I ran this experiment 100 times, how often would I get the a different answer if I stopped now vs. if I ran to completion? The review I linked points out that this is one of those nice cases where it's possible to satisfy Bayesians and Frequentists at the same time though... $\endgroup$ – Matt Krause Jul 11 '13 at 3:06
  • $\begingroup$ But isn't it true that this approach just as easily can run counter to jkndrkn's goals of ending the experiment in a timely manner? Aren't there a fair number of cases where if you actually kept to trying to get an (adjusted) p under $\alpha$ that you'd end up having to collect more data than you would have if you had just conducted the test at the planned n after data collection? $\endgroup$ – russellpierce Jul 11 '13 at 23:51
  • 1
    $\begingroup$ @RussellS.Pierce: I think it depends. Obviously, the curtailment won't, but some other methods do. The O'Brein and Flemming test, for example, can use more data, but can also reject earlier, and Pocock's test even more so. It obviously depends on the specifics: the book linked above has an example with some reasonable values (effect size: 0.25, alpha=0.05, power=0.9, 5 looks). The fixed N version needs 170 subjects to reject; The OBF version needs a max of 180, and the Pocock needs at most 205, but the expected number of subjects is 130 and 117, respectively. $\endgroup$ – Matt Krause Jul 12 '13 at 2:10
1
$\begingroup$

You can stop early, but if you do, your p-values aren't easily interpreted. If you don't care about the interpretation of your p-value, then the way in which the answer to your first two questions are 'no' doesn't matter (too much). Your client seems pragmatic, so the true interpretation of a p-value is probably not a fine point you care about.

I can't speak to the second approach you propose.

However, the first approach is not on solid ground. Normal approximations of binomial distributions aren't valid for proportions that low (which is the method power.prop.test uses, also the method used by Cohen in his classical book on power). Moreover, as far as I am aware, there is no closed form power analysis solution for two-sample proportion tests (cf. How can one perform a two-group binomial power analysis without using normal approximations?). There are however better methods of approximating the confidence intervals of proportions (cf. the package binom). You can use non-overlapping confidence intervals as a partial solution... but this is not the same as estimating a p-value and thus doesn't provide a route to power directly. I hope somebody has a nice closed form solution they will share with the rest of us. If I stumble on one, I'll update the above referenced question. Good luck.

Edit: While I am thinking about it, let me totally pragmatic here for a moment. Your client wants this experiment to end when they are certain that the experimental site is working better than the control site. After you get a decent sample, if you aren't ready to make a decision, just start adjusting the ratio of your random assignment to whatever side is 'winning'. If it was just a blip, regression towards the mean will slip in, you'll become less certain and ease off the ratio. When you are reasonably certain, call it quits and declare a winner. The optimal approach probably would involve Bayesian updating, but I don't know enough about that topic off the top of my head to direct you. However I can assure you that while it may seem counter intuitive at times, the math itself isn't all that hard.

$\endgroup$
0
$\begingroup$

The questions you have are typical questions emerging in statistical tests. There are two 'flavours' of statistics out there, the frequentist and the bayesian. The frequentist answer to both of your questions its easy:

  • NO
  • No, you can't stop early
  • No, you can't measure just longer

Once you defined your setup, you are not alowed to even look at the data (blind analysis). From the frequentist point of view, there is no way around, no cheating no tricks! (EDIT: Of course, there are attempts to do so, and they will also work if used correctly, but most of them are known to introduce biases. )

But there is the bayesian point of view, which is quite different. The bayesian approach needs in contrast to the frequentists an additional input, the a-priori probability distribution. We can call it also previous knowledge or prejudice. Having this, we can use the data/measurement to update our knowledge to the a-posteriori probability. The point is, we can use the data and even more, we can use the data at every intermediate point of the measurement. In each update, the last posterior is our new prior and we can update it with a new measurement to our up-to date knowledge. No early stopping problem at all!

I found a talk discussing quite similar like problems you have and I described above: http://biostat.mc.vanderbilt.edu/wiki/pub/Main/JoAnnAlvarez/BayesianAdaptivePres.pdf

But beside this, are you really sure you need this at all? It seems that you have some system running deciding where to link a request. For this you don't need to proof that your decissions are correct in a statistical sense with a hypothesis test. Have you ever bought a coke, because you could exclude that pepsi is 'right' now with a probability of 95%? It's sufficient to take the one which is just better, not excluding a hypothesis. That would be a trivial algorithm: Calculate uncertainty of rate A, calculate uncertinty of B. Take the difference of both rates and devide it b the uncertainty of the difference. The result is something like the significance of the difference in sigma. Then just take all the links where there is more than two or three sigma difference. Drawback, you will never now if a single decision was statistical correct with some evidence, but in average you will have higher conversion rates.

$\endgroup$
  • 1
    $\begingroup$ I think your first four paragraphs are a bit of a red herring--there are frequentist approaches to interim monitoring. It's true that (Bayesian) posterior updating lends itself nicely to this, but you could frame this problem in many different ways. Thanks for the pointer to the slides! $\endgroup$ – Matt Krause Jul 10 '13 at 18:28
  • $\begingroup$ +1 anyway - the Bayesian approach is probably clear headed here than any frequentist fix. $\endgroup$ – russellpierce Jul 11 '13 at 1:56
  • 2
    $\begingroup$ Eh, there's this...meme...that Bayesian methods let one peek endlessly at the data without any complications. However, the overall Type I error rate isn't actually controlled (why would it be?) and it can get arbitrarily large if after many "looks." You can ameliorate this with an appropriate prior, or you can argue that controlling Type I error is lame, but it's not as if all Bayesian techniques are a panacea. $\endgroup$ – Matt Krause Jul 12 '13 at 2:28
0
$\begingroup$

maybe some methods could be used there like

  • Pocock
  • O’Brien and Flemming
  • Peto

this will adjust the P cutoff based on results and will help wou stop collecting data and economize resources and time.

maybe other works could be added here.

$\endgroup$
  • $\begingroup$ could you provide links or more information for your bullet points? $\endgroup$ – Antoine Jul 10 '16 at 13:57
  • $\begingroup$ i don't have that exact articles because i used a review article that is citing these, i mean approaches are different but i may recommend you taht article that manage the question in medical field : Adaptive sample size modification in clinical trials: start small then ask for more? Christopher Jennisona*† and BruceW. Turnbullb $\endgroup$ – HappyMan Jul 10 '16 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.