I have a typical example of A/B testing where we want to investigate the change in user behaviour after having tested a new website design on a specific group of users. The metric that we're using (and the only one that I have available) it is whether the user opened an account on a competitor website: kind of the opposite of a "conversion rate". The data looks something like this (user is unique)

user design joined_competition_at_least_once
a32 new False
a33 old True

As the difference in the metric is really small between the two groups I've performed a t-test and found that the difference is not significant (p=0.064), which means that the new design likely did not perform better or worse than the old one.

Now, as additional variable I also have the time spent from the introduction of the new design to the last time a user was seen online: this is available for all users, not only the ones that tested the new design.

The question is: can I use this additional info to improve the statistical significance? In theory the time spent on the platform should be a good indicator of whether a user likes it or not, but by comparing the distributions of this variable for the two different groups I haven't seen many differences.

Is there any way to change the test to account for this additional variable or is the first answer already enough?


1 Answer 1


In theory the time spent on the platform should be a good indicator of whether a user likes it or not [...]

In theory, the time spent on a platform can also mean that users have a harder time figuring out how to do things. It all depends on context. In any case, it's moot because you say there's no difference in the time spent.

Looking at the question title, I'm not sure how one can improve the A/B test. If you are referring to the quality of the experiment or its findings, then those are things to think through in design phase. And the phrase "improve statistical significance" implies mining for causal patterns in the data to get findings with p < 0.05. This is bad practice, and should be avoided.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.