I'm a beginner in statistics. When it comes to digital product A/B testing, I'm told by my company's analyst that they typically use the T-test for the hypothesis testing Why would we not use the Z-stat?
I understand that Z-stat is used when we know the population mean & variance and when we want to measure the probability of a sample mean.
I understand the T-test is used when we have smaller sample sizes and we do not know the variance of the population. The two-sample T is used when we want to compare two sample means to each other.
When it comes to digital products, I know the variance of my user base because it's tracked. Wouldn't that mean the Z-stat is able to be applied here? For example, let's say my hypothesis is: "If we insert a widget, then we will increase customer purchase rates"
H0: There is no difference in conversion rates between users with normal site experience (the current population) and site experience with widget.
HA: There is a difference in purchase rates between normal site UX & widget experience.
I know that our current normal site experience has an average customer purchase rate of 2% with a SD of 0.2%. If I take the purchase rate sample mean of the users exposed to the widget site experience, wouldn't I be able to analyze how extreme that mean is against our known population?
Since I know our site's population parameters, why wouldn't I use the Z-stat? What am I missing?
Is it because in the scenario I'm describing above, the population conversion rate is only measuring the people that visited our website before? The T-test would be used to compare two sets of users that never visited the website before?