# Experimental Design of A/B Conversion Rate: How to remedy if daily visitor count is less than desired sample size?

Let's say I have a daily conversion rate of 33.1% with a sample size of 4,500 daily visitors. If I want to run an A/B test and detect a 3% difference in conversion, I would need around 3,881 visitors in each group for a total of 7,762.

Since the desired sample size is greater than the number of daily visitors, I cannot run the test. What ways can I remedy this outside of settling for increasing my minimum detectable difference?

I thought about evaluating weekly conversion rates since I have more unique visitors in a week compared to a day. However, I have some visitors who will convert twice in a week, thus, I believe this invalidates the experiment, right?

I also thought about just running the a/b test for some time and then I can compare the average daily conversion rate of the two groups using a t-test, however, that does not seem to be appropriate either, does it?

Any help/elucidation would be awesome. I've been scanning Cross Validated and other sources for a solution to no avail.

## 1 Answer

Having a visitor visit the site multiple times isn't necessarily a validity issue. Where it becomes a problem is if you are counting the number of visitors in the denominator when computing conversion.

A simple solution is to compute "conversion" with the numerator being the number of visitors to convert once or more over the time period, and the denominator being the number of unique visitors (not visits). If there is not a lot of overlap, you can probably compare pairs of days.

Or, you can compare the number of times visitors converted over a week (that is, performing a t-test, where the values for each visitor are 0, 1, 2, ...).

Unless you have lots of people with multiple visits both approaches will give you very similar results.

A more rigorous approach is to use a panel data model of some kind (e.g., a binary logit model with a random effect for people and lagged effects to deal with learning). I'd definitely be trying the simpler approaches first.

• As a follow up, any particular reason why you would recommend doing a t-test instead of a chi-square test? I hypothesize I would just look at the conversion rates of the two groups over a week (of course counting unique visitors and not visits). Commented Feb 22, 2018 at 0:35