bootstrap resampling technique when comparing medians with finite population size

Question

When comparing differences in samples (e.g., difference in medians) between two groups, I am

1. adjusting group size to account for finite populations of the groups,
2. pooling all of the samples together sampling from the pooled data (with replacement) to create two new groups of data calculating the medians of each group,
3. then the difference of the medians, and
4. using the distribution of the resulting difference to understand the how likely the difference between groups might be observed by random chance

Suppose I'm at a middle school where I give the same lecture to both class 1 and class 2 with respective class sizes of 15 and 20 students. I suspect that class 2 likes the course better since I teach that class after I have had my coffee. To assess attitude between the classes, I survey 5 students in class 1 and 10 students in class 2. The responses from class 1 are {1,2,3,4,5}. The responses from class 2 are {2,3,4,5,6,7,2,3,4,5}. I want to know if the attitude between the two classes taught by this teacher are different, say greater than a certain value x. (In this example, I happen to have ordered categorical responses--say a survey response from 1 to 7--but we can switch this to a continuous variable like grades if needed).

A.C. Davidson and D.V. Hinkley's "bootstrap methods and their applications" provide methods modify sample size when bootstrapping statistics estimating a population quantity, where the population is a known, finite size (pg 92). For example, given a finite population size, we can adjust the resample size upwards to $n′$ where $n′=(n−1)/(1−n/N)$. Here, $N$ is the population size.

Set up and Define the inputs:

import numpy as np
import plotly.graph_objects as go
responses_1 = [1,2,3,4,5] #median is 3
responses_2 = [2,3,4,5,6,7,2,3,4,5] #median is 4
population_size_1 = 15
population_size_2 = 20
sam_pop_ration = len(responses_1)/population_size_1
sam_pop_ration = len(responses_2)/population_size_2

Approach:

def bootstrap_medians_pooled_approach(input_array_1, len_input_array_1, sam_pop_ration_1, \
    input_array_2, len_input_array_2, samp_pop_ration_2, \
    n_resamples):

    #sample 1
    adjusted_n_1 = (len_input_array_1 - 1)/(1 - sam_pop_ratio_1)
    ##some considerations for having a decimal adjusted_n_1 
    base_adjusted_n_1 = int(adjusted_n_1)
    fraction_adjusted_n_1 = adjusted_n_1 - base_adjusted_n_1
    #create an a array of sample 1 resample sizes
    ##alternate size to account for the fraction of adjustment
    adjusted_n_array_1 = [base_adjusted_n_1 + \
        int(np.random.choice([0,1], size = 1, \
        p = [1 - fraction_adjusted_n_1, fraction_adjusted_n_1)) \
        for x in range(n_samples)]
    #sample 2 (same setup as above for sample 1)
    adjusted_n_2 = (len_input_array_2 - 1)/(1 - sam_pop_ratio_2)
    base_adjusted_n_2 = int(adjusted_n_2)
    fraction_adjusted_n_2 = adjusted_n_1 - base_adjusted_n_2
    adjusted_n_array_2 = [base_adjusted_n_2 + \
        int(np.random.choice([0,1], size = 1, \
        p = [1 - fraction_adjusted_n_2, fraction_adjusted_n_2)) \
        for x in range(n_samples)]
    pooled_array_n = np.add(adjusted_n_array_1, adjusted_n_array_2
    pooled_array = input_array_1 + input_array_2

    #create list of resampled medians for group 1 and group 2
    medians_1 = [np.median(np.random.choice(pooled_array, size = x)) \
        for x in adjusted_n_array_1]
    medians_2 = [np.median(np.random.choice(pooled_array, size = x)) \
        for x in adjusted_n_array_2]

n_resamples = 10000
bs_pool_delta = bootstrap_medians_pooled_approach(responses_1, len(responses_1), 
    sam_pop_ratio_1,\
    responses_2, len(responses_2), sam_pop_ratio_2, \
    n_resamples)

#visualize the distribution of deltas results
fig_bsed_pool_deltas = go.Figure()
fig_bsed_pool_deltas.add_trace(go.Histogram(x = bs_pool_delta)

#explore the chance that the observed delta of a given delta might be observed by random chance
deltas = 0.25 * x for x in range(-28,28)
fig_ps_bs = go.Figure()
fig_ps_bs.add_trace(go.Scatter(x = deltas, y = bsed_p_values_pool))

If I wasn't adjusting the group sizes for the finite population, I would shuffle the pooled data into new groups (without replacement). By resampling with replacement, how should I interpret the results? Is it still correct to think about the fig_bsed_pool_deltas as the probability of observing the delta due to random error? Or is this a misapplication of the technique? One thing that bothers me is that I pool the data, but then use the original group size rather than setting the populations of each group to the sum of population_size_1 and population_size_2.

Note: If I don't pool all results but rather 1) resample median values for group 1, 2) resample median values for group 2, and then 3) find the resulting difference vector, I get a much tighter distribution of the delta. I think that I would interpret this as the distribution the the calculated median, not the probability of getting the result by random chance.

Answer 1

One thing that bothers me is that I pool the data, but then use the original group size rather than setting the populations of each group to the sum of population_size_1 and population_size_2.

• The bootstrapping is a way to simulate the effects of the error in the empirical distribution and what it does to a statistic derived from that empirical distribution such as the sample mean or median.

  The effect of increasing the sample size $n^\prime>n$ for the bootstrapping procedure is that it reduces the variation in your bootstrapping samples.

  The reason is because your empirical distribution has a relatively smaller error when you are sampling from a finite distribution.

  (In the extreme case when $n=N$ then you have zero error, and you are supposed to take an infinite sample, which has zero variation)

• The pooling has the effect of allowing an estimate of the emperical distribution that is more accurate. (however, this is only true if the two population distributions are the same.)

---

These two do not interfer so much.

The bootstrapping is simulating variations from sampling a distribution, and ideally that is the real distribution but you are using the empirical distribution instead. If you can use an improvement by using a pooled sample, then that is better.

The adjustment of sample size is made because the bootstrapping – whether it is based on the true distribution, an empirical distribution from pooled groups, or an empirical distribution from a single group – will overestimate the variation in the sampling distribution. This is because the real sampling occurs from a fixed population without replacement and the bootstrapping sampling occurs from a fixed population (that approximates the true population) with replacement.

A reason to use a pooled sample is to improve the approximation of the true population and it does not interfer with the idea of adjusents in the sample size.

---

But note: For a test of equivalence of medians you might choose to not pool the data since the distributions of the two groups can be considered different in other ways.

Especially for small population sizes, one may wonder whether the distributions are the same and whether pooling improves the estimates of the true population. Especially when the sample sizes are close to the population sizes, then the empirical distributions will be good estimates of the true distributions and pooling may not be a good improvement.