1
$\begingroup$

Can we convert any A/B testing problem into a multiple testing problem?

$\endgroup$
3
  • $\begingroup$ What is the context? If you change one button and check for its effect, then you are performing one test. So no, A/B testing is not intrinsically a multiple testing problem. $\endgroup$ Oct 6, 2017 at 0:18
  • 1
    $\begingroup$ @FransRodenburg, why not make that an official answer? Although this question is rather sparse, it could be answered, & that would be it, AFAICT. $\endgroup$ Oct 6, 2017 at 1:13
  • $\begingroup$ Thank you for the suggestion, I was not sure it would be sufficient information for an answer. $\endgroup$ Oct 6, 2017 at 1:17

1 Answer 1

3
$\begingroup$

That depends on the context of using A/B testing:

If you change one button and check for its effect, then you are performing one test. So A/B testing is not intrinsically a multiple testing problem.

$\endgroup$
4
  • $\begingroup$ Ok. If we were to perform multiple A/B testing, potentially with a large number of features, would that be intrinsically a multiple testing problem? $\endgroup$
    – Jack
    Oct 6, 2017 at 17:08
  • $\begingroup$ Or in the case where companies run a sequence of A/B tests over time. $\endgroup$
    – Jack
    Oct 6, 2017 at 18:59
  • $\begingroup$ If you test $p$ features from a single experiment for significance, then you need to correct for $p$ tests. You could think of Bonferroni, or FDR if the former is too conservative. If you perform a new A/B test with new observations, then no correction is applied for having performed one previously on a different dataset. $\endgroup$ Oct 6, 2017 at 23:01
  • $\begingroup$ Thanks. I'm currently working on a online FDR control paper, that's why I'm thinking of a multiple A/B testing application. $\endgroup$
    – Jack
    Oct 7, 2017 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.