# Are A/B testing problems intrinsically a multiple testing problem?

Can we convert any A/B testing problem into a multiple testing problem?

• What is the context? If you change one button and check for its effect, then you are performing one test. So no, A/B testing is not intrinsically a multiple testing problem. – Frans Rodenburg Oct 6 '17 at 0:18
• @FransRodenburg, why not make that an official answer? Although this question is rather sparse, it could be answered, & that would be it, AFAICT. – gung - Reinstate Monica Oct 6 '17 at 1:13
• Thank you for the suggestion, I was not sure it would be sufficient information for an answer. – Frans Rodenburg Oct 6 '17 at 1:17

• If you test $p$ features from a single experiment for significance, then you need to correct for $p$ tests. You could think of Bonferroni, or FDR if the former is too conservative. If you perform a new A/B test with new observations, then no correction is applied for having performed one previously on a different dataset. – Frans Rodenburg Oct 6 '17 at 23:01