# Digital Product A/B Testing: Z vs T test

I'm a beginner in statistics. When it comes to digital product A/B testing, I'm told by my company's analyst that they typically use the T-test for the hypothesis testing Why would we not use the Z-stat?

I understand that Z-stat is used when we know the population mean & variance and when we want to measure the probability of a sample mean.

I understand the T-test is used when we have smaller sample sizes and we do not know the variance of the population. The two-sample T is used when we want to compare two sample means to each other.

When it comes to digital products, I know the variance of my user base because it's tracked. Wouldn't that mean the Z-stat is able to be applied here? For example, let's say my hypothesis is: "If we insert a widget, then we will increase customer purchase rates"

H0: There is no difference in conversion rates between users with normal site experience (the current population) and site experience with widget.

HA: There is a difference in purchase rates between normal site UX & widget experience.

I know that our current normal site experience has an average customer purchase rate of 2% with a SD of 0.2%. If I take the purchase rate sample mean of the users exposed to the widget site experience, wouldn't I be able to analyze how extreme that mean is against our known population?

Since I know our site's population parameters, why wouldn't I use the Z-stat? What am I missing?

Is it because in the scenario I'm describing above, the population conversion rate is only measuring the people that visited our website before? The T-test would be used to compare two sets of users that never visited the website before?

• You know the sample variance. You do not know the population variance. And if you do know the population variance, how did you determine that without finding the population mean? But I’d ask this very question to your company’s analyst. Maybe even report her answer here.
– Dave
Apr 27, 2020 at 17:08

• Not quite. The z-test is the right approach, but not because we have population level information. The z test is the right approach because we don't need to compute a separate estimate of the variability in our measure. We get it for free when we compute the sample proportion $\hat{p}$ because $\hat{\sigma}^2 = \hat{p}(1-\hat{p})/n$ Apr 27, 2020 at 19:32