1
$\begingroup$

The Welch t-test is best used when we cannot make an equal variance assumption between our treatment and control groups (our two samples).

However, in A/B testing, it's not clear to me how we could know the variance of the two samples ahead of time. We could, of course, assume they're equal: But how could we be sure? It seems very probable that a new feature could introduce more variance in a treatment statistic.

Therefore, do we need to always use a Welch t-test, barring some strong assumptions or a non-stat sig F-test, when A/B testing? This also makes me wonder how we can actually power a test if we won't know the variance of the treatment until test time.

$\endgroup$
3

0