1
$\begingroup$

Let’s say there is a new product feature “X” that is going to replace the current product feature “Y”. The business wants to make a decision tto implement “X” only if it performs better than “Y”.

Here's my hypothesis for this experiment

H0: There is no change in conversion rate between users who were given “Y” and users who were given new feature “X”

H1: there is an increase in conversion rate for users who were given “X” compared to users who were given “Y”

The question is should I be using a 1-tailed or 2-tailed test for this? And why?


These are my answers and would like your feedback on them

I should be using a 1-tailed test (right-tailed) because it can help us reach a stat sig result that feature X is better than the control. If I am concern that this feature may have adverse effect on the conversion rate, i would be better off conducting another 1-tailed test (left-tailed) to determine it.

We should not be using a 2-tailed test because 2-tailed test can only help us reach a stat sig result that there is no difference between feature X and feature Y. I could not say that it performed better or worst.

I am asking this question because there are many articles online that advocate the use of 2-tailed test for this kind of experiment but there is also this article that debunks it. http://blog.analytics-toolkit.com/2017/one-tailed-two-tailed-tests-significance-ab-testing/

$\endgroup$

2 Answers 2

2
$\begingroup$

You have no interest in detecting if the new product performs worse than the old product, only of it performs better. Since you know this before you look at the data, you increase your power to detect a difference by performing a one sided test. This is at the expense of being able to detect inferiority, but you have made the business decision that such a change is not of interest.

Had you looked at the data, noticed the new product to perform better, and decided to do a one-sided test, this would be a mistake. You make the one-tailed vs two-tailed decision before you look at your data.

$\endgroup$
1
$\begingroup$

I am not able to comment on your answers. So here is an analogy.

So imagine you were doing a shuttle run in a park. You reach a corner bend down to touch the ground and you run back again to the corner where you started from and bend down to touch the ground and do it several times. After your running session you realise that you have lost your ring. You are pretty sure that you have lost it sometime when you bent to touch the ground. But unfortunately the sun has set and it has turned dark. But luckily, You have a couple torch lights at your disposal. When you are sure about which corner you lost the ring you use both the torch lights to trace the ring in the same corner (More power to find the ring if it was lost in that particular corner) that's a one tiled test. But when you are not sure about which corner you lost the ring, you use one torch light for each corner (Less power in each corner but you are covering both the corners). That is a two tiled test. It's not that two tiled test won't help you find the direction of effect. It will but with lesser power. When in doubt always use a two-tailed test. Hope you find your ring!

$\endgroup$
1
  • $\begingroup$ thanks for the analogy above! i wished i can select this as answer too.you pointed out a very good point here. the considerations that i have to take into account is if i want to commit to an experiment (2-tailed) that requires 20%-60% more users but could only detect he direction of effect with lesser power. i guess it all depends on the context of the business decision and the availability of users. thanks! $\endgroup$ Commented Feb 23, 2021 at 3:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.