0
$\begingroup$

Suppose a car rental company historically only rented white cars, and they want to to test if red cars make customers more likely to pay for an upgrade, such as enabling satellite radio. (I am just using this as a simple example of an AB test with a binary outcome.)

The car rental company's data suggests that base rate of upgrades was 30% in the year leading up to the test.

In order to detect a 1% point increase in the success rate I would need to sample 33,274 renters per car color condition according to the power.prop.test() function in R, which--I believe--calculates variance based on the binomial distribution. This is appropriate when sampling with replacement or when sampling without replacement a small proportion of a large population.

power.prop.test(p1 = .3, p2 = .31, power = .8, sig.level = .05)

But what if this is a local car rental service that effectively only rents to people in a 50 mile radius and there are only 1000 eligible renters in that area? Furthermore in a typical month only 100 customers rent a car.

I will never reach 33,274 renters per car color condition, but I could sample the entire population of eligible customers long before approaching 33,274 renters. In a scenario like this, wouldn't I be justified in assuming a hypergeometric distribution to estimate the variance of the success rates in my red and white car conditions if I sample without replacement a large proportion of the eligible population?

Further I am wondering if I ran the test for a month should treat 1000 (N of all eligible customers) or 100 (N of eligible monthly traffic) as the population size when calculating variance based on the hypergeometric distribution.

# hypergeometric variance when N = 100
n <- 50 
N <- 100
k <- .3 * N
# formula for hypergeometric variance per https://stattrek.com/probability-distributions/hypergeometric.aspx        
n * k * ( N - k ) * ( N - n ) / ( N^2 * ( N - 1 ) )

Hypergeometric variance = 5.3 when sampling 50 customers from a population of 100.

# hypergeometric variance when N = 1000.
n <- 50 
N <- 1000
k <- .3 * N
# formula for hypergeometric variance per https://stattrek.com/probability-distributions/hypergeometric.aspx        
n * k * (N - k) * (N - n) / ( N^2 * (N - 1) )

Hypergeometric variance = 10.0 when sampling 50 customers from a population of 1000.

# binomial variance
n <- 50
p <- .3 
# formula for binomial variance
n * p * (1 - p)

Binomial variance = 10.5 when sampling 50 customers.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ A rough, but commonly used, rule of thumb is to use binomial when the sample size is less than 10% (maybe 20%) of the population size. For power and sample size analysis, it may be a good idea to find sample size based on binomial computations, then use hypergeometric if that seems necessary. Depending on the software you're using, there may not be a 'built-in' procedure with hypergeometric. If not, you can get close with simulation. $\endgroup$
    – BruceET
    Apr 28, 2021 at 20:30
  • $\begingroup$ Thank you @BruceET, I have found bootstrapped simulations useful in this once I have assumed a certain value for the finite population, but what I am less sure about is where to draw the line for the size of the finite population in a scenario like the one above. $\endgroup$
    – Joe
    Apr 28, 2021 at 20:33

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.