Disclosure: Stat noob trying to understand the basics of ab testing

Context: I'm running an AB test with 3 treatment test to see which one yields the highest click thru rate.

Q: What is the recommend hypothesis test for this experiment.

Option 1) t-test which tests equality between the control and one variant for each pair: control-variant1 , control-variant2, control-variant3

Option 2) ANOVA test which will test for equality among all variants.

Option 3) Both t test and anova? Not sure that makes sense but just wondering if some have ever done that. Let's say I run a t test between control and variant 1 and determine statistical significance for p < 0.05 then automatically running an ANOVA test should yield the same conclusion since at least one variant being different allows to reject the NULL hypothesis

Option 4) Any other ????

  • $\begingroup$ What hypothesis or hypotheses do you want to test? In other words, what questions do you want to answer? $\endgroup$
    – Dave
    Commented May 21, 2022 at 16:05
  • $\begingroup$ @Dave Which variant is performing the best $\endgroup$ Commented May 21, 2022 at 16:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.