Can we convert any A/B testing problem into a multiple testing problem?
3
-
$\begingroup$ What is the context? If you change one button and check for its effect, then you are performing one test. So no, A/B testing is not intrinsically a multiple testing problem. $\endgroup$– Frans RodenburgCommented Oct 6, 2017 at 0:18
-
1$\begingroup$ @FransRodenburg, why not make that an official answer? Although this question is rather sparse, it could be answered, & that would be it, AFAICT. $\endgroup$– gung - Reinstate MonicaCommented Oct 6, 2017 at 1:13
-
$\begingroup$ Thank you for the suggestion, I was not sure it would be sufficient information for an answer. $\endgroup$– Frans RodenburgCommented Oct 6, 2017 at 1:17
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
That depends on the context of using A/B testing:
If you change one button and check for its effect, then you are performing one test. So A/B testing is not intrinsically a multiple testing problem.
answered Oct 6, 2017 at 1:16
-
$\begingroup$ Ok. If we were to perform multiple A/B testing, potentially with a large number of features, would that be intrinsically a multiple testing problem? $\endgroup$– JackCommented Oct 6, 2017 at 17:08
-
$\begingroup$ Or in the case where companies run a sequence of A/B tests over time. $\endgroup$– JackCommented Oct 6, 2017 at 18:59
-
$\begingroup$ If you test $p$ features from a single experiment for significance, then you need to correct for $p$ tests. You could think of Bonferroni, or FDR if the former is too conservative. If you perform a new A/B test with new observations, then no correction is applied for having performed one previously on a different dataset. $\endgroup$ Commented Oct 6, 2017 at 23:01
-
$\begingroup$ Thanks. I'm currently working on a online FDR control paper, that's why I'm thinking of a multiple A/B testing application. $\endgroup$– JackCommented Oct 7, 2017 at 0:17