1
$\begingroup$

I want to compare a percentile metric for control vs treatment. Using bootstrap, I see two ways of doing so and wonder which one makes more sense.

Approach 1: Every time:

  1. Bootstrap the control group and get the percentile,
  2. Bootstrap the treatment group and get the percentile
  3. Compute the difference. Repeat B times to get the differences and get the CI.

Approach 2:

  1. Bootstrap the control B times, get the average and variance of the metric
  2. Bootstrap the treatment B times, get the average and variance of the metric
  3. Compute the difference and its variance, assuming they are independent.

Below is a simulation code in R. The 2nd approach has narrower CI, but requires normal assumption.

A <- rnorm(1000, 0.1, 1)
B <- rnorm(1000, 0.15, 1)

pctA = quantile(A, 0.99)
pctB = quantile(B, 0.99)

# Approach 1:
# 1. Bootstrap each set
# 2. Compute the differences for each bootstrap sample
# 3. Get confidence intervals
N = 500
diffBoot = rep(NA, N)
for (i in 1:N) {
    tempA = sample(A, size=1000, replace=TRUE)
    tempB = sample(B, size=1000, replace=TRUE) 
    diff = quantile(tempA, 0.99) - quantile(tempB, 0.99)
    diffBoot[i] = diff
}
mean(diffBoot)
quantile(diffBoot, c(0.025, 0.9725))


# Approach 2: 
# 1. Bootstrap each set
# 2. Compute metrics and variances
# 3. Get differences and confidence intervals
N = 500
pctAboot = rep(NA, N)
pctBboot = rep(NA, N)
for (i in 1:N) {
    tempA = sample(A, size=1000, replace=TRUE)
    tempB = sample(B, size=1000, replace=TRUE) 
    pctAtemp = quantile(tempA, 0.99) 
    pctBtemp = quantile(tempB, 0.99)
    pctAboot[i] = pctAtemp
    pctBboot[i] = pctBtemp
}
meanboot <- mean(pctAboot - pctBboot)
sdboot <- sqrt(var(pctAboot) + var(pctBboot))
meanboot / (1.96 * sdboot)
```
$\endgroup$
2
  • $\begingroup$ Your code seems to have some typo error, tempB = sample(B, size=1000, replace=TRUE): here B is 500, not B <- rnorm(1000, 0.15, 1) $\endgroup$
    – wei
    Commented Apr 23, 2023 at 3:26
  • $\begingroup$ Thank you. It should be fixed now. $\endgroup$
    – user28363
    Commented Apr 23, 2023 at 4:05

1 Answer 1

0
$\begingroup$

If we calculate the statistics of approach 1 and approach 2 condition on the same sample (this can reduce the fluctuation of the simulation itself), the two approaches will give the same diff estimator and very close variance estimation, while the difference in variance is due to the random correlation between pctAtemp and pctBtemp(pctAtemp and pctBtemp is independent).

set.seed(1)
sample_cnt = 1000
tempA = rnorm(sample_cnt, 0.10, 1)
tempB = rnorm(sample_cnt, 0.15, 1)

N <- 500 ## this give  
p <- 0.99 ## test percentile
diffBoot <- rep(NA, N)
pctAboot <- rep(NA, N)
pctBboot <- rep(NA, N)
for (b in 1:N) {
  tempA = sample(A, size = sample_cnt, replace = TRUE)
  tempB = sample(B, size = sample_cnt, replace = TRUE)
  # tempA = rnorm(sample_cnt, 0.10, 1)
  # tempB = rnorm(sample_cnt, 0.15, 1)
  pctAtemp = quantile(tempA, p)
  pctBtemp = quantile(tempB, p)
  
  ## For each sample, two approach statistics are calculated simultaneously, which can reduce the variance of the simulation
  diffBoot[b] = pctAtemp - pctBtemp
  pctAboot[b] = pctAtemp
  pctBboot[b] = pctBtemp
}
est1 <- mean(diffBoot)
var1 <- var(diffBoot)
sprintf("approach 1 est: %.6f, var: %.6f", est1, var1)

est2 <- mean(pctAboot - pctBboot)
var2 <- var(pctAboot) + var(pctBboot)
sprintf("approach 2 est: %.6f, var: %.6f", est2, var2)

sprintf("var diff(%.6f) is var(pctAtemp, pctBtemp) * 2 = %.6f", var2 - var1, var(pctAboot, pctBboot) * 2)

[1] "approach 1 est: -0.009805, var: 0.025579"
[1] "approach 2 est: -0.009805, var: 0.024857"
[1] "var diff(-0.000722) is var(pctAtemp, pctBtemp) * 2 = -0.000722"

Another difference is: approach 1 is close to percentile bootstrap CI, approach 2 is closer to Standard interval bootstrap.Is it true that the percentile bootstrap should never be used?

$\endgroup$
1
  • $\begingroup$ Thanks for the answer. And the extra link is also very informative. Appreciate that! $\endgroup$
    – user28363
    Commented Apr 23, 2023 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.