0
$\begingroup$

Could someone help me understand which statistical test should be used to test for the statistical significance between these rates?

Our group has implemented a new process change which has resulted in a change in a key metric which is measured in a proportion (I.e conversion rate). I want to find out whether or not this change is statistically significant against the historical baseline rate ( prior to implementation). I’m uncertain if I should be using a one proportion test to test this new rate against the baseline. Or is a chi squared test more appropriate in this case? Should I be looking at these as two different populations because of the change? Hopefully my question makes sense.

$\endgroup$
1
  • $\begingroup$ Your question makes sense, and it sounds like you are thinking about the situation correctly. "Historical baseline" sounds like it suggests a test of counts in one proportion against a theoretical proportion, like a binomial test or a chi-square goodness-of-fit. $\endgroup$ Feb 27, 2020 at 1:55

2 Answers 2

1
$\begingroup$

If you already have an estimate of the conversion prior to the intervention, a single sample test is fine. Your null would just be that $p=p_c$, where $p_c$ is the conversion in the prior case.

If you need to estimate the prior conversion rate then a 2 sample is needed.

Design of the experiment is going to be really important. If you include the same users in each group, correlation will be an issue. Think more about the design of the experiment and the test should naturally follow.

$\endgroup$
1
  • $\begingroup$ Thank you Demetri. Yes, we have the conversion prior to the intervention, and it is just cumulative historical prior to the intervention. Essentially this is just a key performance indicator metric we have been continuously monitoring and improving. And once the process implementation goes through, we will not revert back. So the new rate after the change will now become our next baseline for the next change. So it’s not truly an ‘experiment’. But we do want to measure the new conversion rate as a ‘test’ group. $\endgroup$
    – StealthEXE
    Feb 27, 2020 at 23:03
0
$\begingroup$

Generally speaking, you would want to do a 2-proportion test, because the baseline itself is a sample, and you want to know if your recent sample is statistically different from your baseline sample. Using a 1-proportion test will tell you if your recent sample is different from the baseline value, but that won't tell you if your process output has changed. You need the 2-proportion test for that.

For the 2-proportion test, however, I would not use chi-squared. I would use the exact test, based on the binomial distribution. Your stat package should offer a specific 2-proportion test. I generally prefer the result given for Fisher's Exact Test, unless a confidence interval is needed, in which case you need the asymptotic result.

Of course, to do the 2-proportion test requires that you have the numerator and denominator for the historical baseline. If all you have is a stated rate, then you're kind of stuck with the 1-proportion test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.