1
$\begingroup$

The Welch t-test is best used when we cannot make an equal variance assumption between our treatment and control groups (our two samples).

However, in A/B testing, it's not clear to me how we could know the variance of the two samples ahead of time. We could, of course, assume they're equal: But how could we be sure? It seems very probable that a new feature could introduce more variance in a treatment statistic.

Therefore, do we need to always use a Welch t-test, barring some strong assumptions or a non-stat sig F-test, when A/B testing? This also makes me wonder how we can actually power a test if we won't know the variance of the treatment until test time.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You may not know the variance under this particular treatment but there could have been previous studies on similar treatments to give you an idea what the variance may be. If that's not the case or you don't want to draw such parallels, you can run a pilot study. $\endgroup$
    – dipetkov
    Commented Oct 8, 2023 at 10:52
  • $\begingroup$ There are lots of related threads on CV that may be helpful to review, eg. Power calculation for two-sample Welch's t test Behaviour of Welch's t-test with unequal group sizes, Welch test seems to perform much worse than equal variance t-test $\endgroup$
    – dipetkov
    Commented Oct 8, 2023 at 11:05
  • 2
    $\begingroup$ The Bayesian t-test has major advantages of not lulling us into thinking dichotomously. Adding 2 parameters (for the variance ratio and for the degree of non-normality) takes into account what we don’t know, and relaxes assumptions as the sample size increases. See here. $\endgroup$
    – Frank Harrell
    Commented Oct 8, 2023 at 12:14

0

Reset to default