Improving A/B test with additional variable (time spent)

Question

I have a typical example of A/B testing where we want to investigate the change in user behaviour after having tested a new website design on a specific group of users. The metric that we're using (and the only one that I have available) it is whether the user opened an account on a competitor website: kind of the opposite of a "conversion rate". The data looks something like this (user is unique)

user	design	joined_competition_at_least_once
a32	new	False
a33	old	True

As the difference in the metric is really small between the two groups I've performed a t-test and found that the difference is not significant (p=0.064), which means that the new design likely did not perform better or worse than the old one.

Now, as additional variable I also have the time spent from the introduction of the new design to the last time a user was seen online: this is available for all users, not only the ones that tested the new design.

The question is: can I use this additional info to improve the statistical significance? In theory the time spent on the platform should be a good indicator of whether a user likes it or not, but by comparing the distributions of this variable for the two different groups I haven't seen many differences.

Is there any way to change the test to account for this additional variable or is the first answer already enough?

Answer 1

In theory the time spent on the platform should be a good indicator of whether a user likes it or not [...]

In theory, the time spent on a platform can also mean that users have a harder time figuring out how to do things. It all depends on context. In any case, it's moot because you say there's no difference in the time spent.

Looking at the question title, I'm not sure how one can improve the A/B test. If you are referring to the quality of the experiment or its findings, then those are things to think through in design phase. And the phrase "improve statistical significance" implies mining for causal patterns in the data to get findings with p < 0.05. This is bad practice, and should be avoided.