2
$\begingroup$

Let's say I have a daily conversion rate of 33.1% with a sample size of 4,500 daily visitors. If I want to run an A/B test and detect a 3% difference in conversion, I would need around 3,881 visitors in each group for a total of 7,762.

Since the desired sample size is greater than the number of daily visitors, I cannot run the test. What ways can I remedy this outside of settling for increasing my minimum detectable difference?

I thought about evaluating weekly conversion rates since I have more unique visitors in a week compared to a day. However, I have some visitors who will convert twice in a week, thus, I believe this invalidates the experiment, right?

I also thought about just running the a/b test for some time and then I can compare the average daily conversion rate of the two groups using a t-test, however, that does not seem to be appropriate either, does it?

Any help/elucidation would be awesome. I've been scanning Cross Validated and other sources for a solution to no avail.

$\endgroup$
0
$\begingroup$

Having a visitor visit the site multiple times isn't necessarily a validity issue. Where it becomes a problem is if you are counting the number of visitors in the denominator when computing conversion.

A simple solution is to compute "conversion" with the numerator being the number of visitors to convert once or more over the time period, and the denominator being the number of unique visitors (not visits). If there is not a lot of overlap, you can probably compare pairs of days.

Or, you can compare the number of times visitors converted over a week (that is, performing a t-test, where the values for each visitor are 0, 1, 2, ...).

Unless you have lots of people with multiple visits both approaches will give you very similar results.

A more rigorous approach is to use a panel data model of some kind (e.g., a binary logit model with a random effect for people and lagged effects to deal with learning). I'd definitely be trying the simpler approaches first.

$\endgroup$
  • $\begingroup$ As a follow up, any particular reason why you would recommend doing a t-test instead of a chi-square test? I hypothesize I would just look at the conversion rates of the two groups over a week (of course counting unique visitors and not visits). $\endgroup$ – grantog Feb 22 '18 at 0:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.