# Digital Product A/B Testing: Z vs T test

I'm a beginner in statistics. When it comes to digital product A/B testing, I'm told by my company's analyst that they typically use the T-test for the hypothesis testing Why would we not use the Z-stat?

I understand that Z-stat is used when we know the population mean & variance and when we want to measure the probability of a sample mean.

I understand the T-test is used when we have smaller sample sizes and we do not know the variance of the population. The two-sample T is used when we want to compare two sample means to each other.

When it comes to digital products, I know the variance of my user base because it's tracked. Wouldn't that mean the Z-stat is able to be applied here? For example, let's say my hypothesis is: "If we insert a widget, then we will increase customer purchase rates"

H0: There is no difference in conversion rates between users with normal site experience (the current population) and site experience with widget.

HA: There is a difference in purchase rates between normal site UX & widget experience.

I know that our current normal site experience has an average customer purchase rate of 2% with a SD of 0.2%. If I take the purchase rate sample mean of the users exposed to the widget site experience, wouldn't I be able to analyze how extreme that mean is against our known population?

Since I know our site's population parameters, why wouldn't I use the Z-stat? What am I missing?

Is it because in the scenario I'm describing above, the population conversion rate is only measuring the people that visited our website before? The T-test would be used to compare two sets of users that never visited the website before?

• You know the sample variance. You do not know the population variance. And if you do know the population variance, how did you determine that without finding the population mean? But I’d ask this very question to your company’s analyst. Maybe even report her answer here. – Dave Apr 27 '20 at 17:08

So I assume you are talking about binary experiments (e.g. convert not convert). In that case you are right that a z-test is appropriate, but for the wrong reasons.

Student's t-test is used when comparing means because, as you've noted, the estimation of the standard deviation leads to excess uncertainty. However, in a z-test our estimate of the standard deviation is a function of the sample mean.

When it comes to digital products, I know the variance of my user base because it's tracked.

Might that variance change in a month? Or when new users are added? Or given some world event? When you say you know the population variance, your definition of "population" is implicitly conditioning on a whole bunch of unspoken assumptions. So, even though you track the variance of your metric (be it conversion, or clicks, or money spent on your app) it is still safer to soak up some variability by using the t-test than by assuming you have perfect information on the population.

• This makes a lot of sense! So it's possible to use a Z-test in this situation but it's not the best tool due to assumptions about the population. – Michael Ninh Apr 27 '20 at 18:20
• Not quite. The z-test is the right approach, but not because we have population level information. The z test is the right approach because we don't need to compute a separate estimate of the variability in our measure. We get it for free when we compute the sample proportion $\hat{p}$ because $\hat{\sigma}^2 = \hat{p}(1-\hat{p})/n$ – Demetri Pananos Apr 27 '20 at 19:32