10
$\begingroup$

I am a software developer looking to develop an alternative for the simple hypothesis testing scheme described here. In short, the test works as follows:

  • Two URLs are compared for their ability to convert visitors.
  • Discrete samples are captured. Each of them either converts or fails to do so. A conversion is a desirable event.
  • A higher conversion rate is seen as desirable.

I am looking into SPRT because while the users of my testing system are comfortable setting significance level and power ahead of time, they are not comfortable setting sample size ahead of time. They want a test with a stopping rule rather than one that has to capture a specific sample size.

I understand most of the math in the linked Wikipedia article. However, I don't understand how to compute the likelihood ratio. Could someone provide a concrete example of how to compute the likelihood ratio and $S_{i}$ for the following events during the example test:

  • 0.95 significance level
  • 0.80 power
  • Two URLs: URL1 and URL2

Events:

  1. Visitor arrives at URL1. Visitor converts.
  2. Visitor arrives at URL1. Visitor fails to convert.
  3. Visitor arrives at URL2. Visitor fails to convert.
  4. Visitor arrives at URL1. Visitor fails to covnert.
  5. Visitor arrives at URL2. Visitor converts.
  6. Visitor arrives at URL2. Visitor converts.

If you could write out the likelihood ratio and $S_{i}$ at each step I would appreciate it!

$\endgroup$
1
  • $\begingroup$ Evan Miller wrote several articles on A/B testing, including Simple Sequential A/B Testing which comes with a list of useful references. (Sequential testing with likelihood ratios is discussed in one of those references.) $\endgroup$ – chl Nov 18 '20 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.