Suppose a car rental company historically only rented white cars, and they want to to test if red cars make customers more likely to pay for an upgrade, such as enabling satellite radio. (I am just using this as a simple example of an AB test with a binary outcome.)
The car rental company's data suggests that base rate of upgrades was 30% in the year leading up to the test.
In order to detect a 1% point increase in the success rate I would need to sample 33,274 renters per car color condition according to the
power.prop.test() function in R, which--I believe--calculates variance based on the binomial distribution. This is appropriate when sampling with replacement or when sampling without replacement a small proportion of a large population.
power.prop.test(p1 = .3, p2 = .31, power = .8, sig.level = .05)
But what if this is a local car rental service that effectively only rents to people in a 50 mile radius and there are only 1000 eligible renters in that area? Furthermore in a typical month only 100 customers rent a car.
I will never reach 33,274 renters per car color condition, but I could sample the entire population of eligible customers long before approaching 33,274 renters. In a scenario like this, wouldn't I be justified in assuming a hypergeometric distribution to estimate the variance of the success rates in my red and white car conditions if I sample without replacement a large proportion of the eligible population?
Further I am wondering if I ran the test for a month should treat 1000 (N of all eligible customers) or 100 (N of eligible monthly traffic) as the population size when calculating variance based on the hypergeometric distribution.
# hypergeometric variance when N = 100 n <- 50 N <- 100 k <- .3 * N # formula for hypergeometric variance per https://stattrek.com/probability-distributions/hypergeometric.aspx n * k * ( N - k ) * ( N - n ) / ( N^2 * ( N - 1 ) )
Hypergeometric variance = 5.3 when sampling 50 customers from a population of 100.
# hypergeometric variance when N = 1000. n <- 50 N <- 1000 k <- .3 * N # formula for hypergeometric variance per https://stattrek.com/probability-distributions/hypergeometric.aspx n * k * (N - k) * (N - n) / ( N^2 * (N - 1) )
Hypergeometric variance = 10.0 when sampling 50 customers from a population of 1000.
# binomial variance n <- 50 p <- .3 # formula for binomial variance n * p * (1 - p)
Binomial variance = 10.5 when sampling 50 customers.