Timeline for Are A/B testing problems intrinsically a multiple testing problem?
Current License: CC BY-SA 3.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 7, 2017 at 0:17 | comment | added | Jack | Thanks. I'm currently working on a online FDR control paper, that's why I'm thinking of a multiple A/B testing application. | |
| Oct 7, 2017 at 0:16 | vote | accept | Jack | ||
| Oct 6, 2017 at 23:01 | comment | added | Frans Rodenburg | If you test $p$ features from a single experiment for significance, then you need to correct for $p$ tests. You could think of Bonferroni, or FDR if the former is too conservative. If you perform a new A/B test with new observations, then no correction is applied for having performed one previously on a different dataset. | |
| Oct 6, 2017 at 19:56 | vote | accept | Jack | ||
| Oct 6, 2017 at 19:56 | |||||
| Oct 6, 2017 at 18:59 | comment | added | Jack | Or in the case where companies run a sequence of A/B tests over time. | |
| Oct 6, 2017 at 17:08 | comment | added | Jack | Ok. If we were to perform multiple A/B testing, potentially with a large number of features, would that be intrinsically a multiple testing problem? | |
| Oct 6, 2017 at 1:16 | history | answered | Frans Rodenburg | CC BY-SA 3.0 |