I have a typical example of A/B testing where we want to investigate the change in user behaviour after having tested a new website design on a specific group of users. The metric that we're using (and the only one that I have available) it is whether the user opened an account on a competitor website: kind of the opposite of a "conversion rate". The data looks something like this (user is unique)
As the difference in the metric is really small between the two groups I've performed a t-test and found that the difference is not significant (p=0.064), which means that the new design likely did not perform better or worse than the old one.
Now, as additional variable I also have the time spent from the introduction of the new design to the last time a user was seen online: this is available for all users, not only the ones that tested the new design.
The question is: can I use this additional info to improve the statistical significance? In theory the time spent on the platform should be a good indicator of whether a user likes it or not, but by comparing the distributions of this variable for the two different groups I haven't seen many differences.
Is there any way to change the test to account for this additional variable or is the first answer already enough?