1
$\begingroup$

Can we convert any A/B testing problem into a multiple testing problem?

$\endgroup$
3
  • $\begingroup$ What is the context? If you change one button and check for its effect, then you are performing one test. So no, A/B testing is not intrinsically a multiple testing problem. $\endgroup$ – Frans Rodenburg Oct 6 '17 at 0:18
  • 1
    $\begingroup$ @FransRodenburg, why not make that an official answer? Although this question is rather sparse, it could be answered, & that would be it, AFAICT. $\endgroup$ – gung - Reinstate Monica Oct 6 '17 at 1:13
  • $\begingroup$ Thank you for the suggestion, I was not sure it would be sufficient information for an answer. $\endgroup$ – Frans Rodenburg Oct 6 '17 at 1:17
3
$\begingroup$

That depends on the context of using A/B testing:

If you change one button and check for its effect, then you are performing one test. So A/B testing is not intrinsically a multiple testing problem.

$\endgroup$
4
  • $\begingroup$ Ok. If we were to perform multiple A/B testing, potentially with a large number of features, would that be intrinsically a multiple testing problem? $\endgroup$ – Jack Oct 6 '17 at 17:08
  • $\begingroup$ Or in the case where companies run a sequence of A/B tests over time. $\endgroup$ – Jack Oct 6 '17 at 18:59
  • $\begingroup$ If you test $p$ features from a single experiment for significance, then you need to correct for $p$ tests. You could think of Bonferroni, or FDR if the former is too conservative. If you perform a new A/B test with new observations, then no correction is applied for having performed one previously on a different dataset. $\endgroup$ – Frans Rodenburg Oct 6 '17 at 23:01
  • $\begingroup$ Thanks. I'm currently working on a online FDR control paper, that's why I'm thinking of a multiple A/B testing application. $\endgroup$ – Jack Oct 7 '17 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.