When comparing differences in samples (e.g., difference in medians) between two groups, I am
- adjusting group size to account for finite populations of the groups,
- pooling all of the samples together sampling from the pooled data (with replacement) to create two new groups of data calculating the medians of each group,
- then the difference of the medians, and
- using the distribution of the resulting difference to understand the how likely the difference between groups might be observed by random chance
Suppose I'm at a middle school where I give the same lecture to both class 1 and class 2 with respective class sizes of 15 and 20 students. I suspect that class 2 likes the course better since I teach that class after I have had my coffee. To assess attitude between the classes, I survey 5 students in class 1 and 10 students in class 2. The responses from class 1 are {1,2,3,4,5}. The responses from class 2 are {2,3,4,5,6,7,2,3,4,5}. I want to know if the attitude between the two classes taught by this teacher are different, say greater than a certain value x. (In this example, I happen to have ordered categorical responses--say a survey response from 1 to 7--but we can switch this to a continuous variable like grades if needed).
A.C. Davidson and D.V. Hinkley's "bootstrap methods and their applications" provide methods modify sample size when bootstrapping statistics estimating a population quantity, where the population is a known, finite size (pg 92). For example, given a finite population size, we can adjust the resample size upwards to $n′$ where $n′=(n−1)/(1−n/N)$. Here, $N$ is the population size.
Set up and Define the inputs:
import numpy as np
import plotly.graph_objects as go
responses_1 = [1,2,3,4,5] #median is 3
responses_2 = [2,3,4,5,6,7,2,3,4,5] #median is 4
population_size_1 = 15
population_size_2 = 20
sam_pop_ration = len(responses_1)/population_size_1
sam_pop_ration = len(responses_2)/population_size_2
Approach:
def bootstrap_medians_pooled_approach(input_array_1, len_input_array_1, sam_pop_ration_1, \
input_array_2, len_input_array_2, samp_pop_ration_2, \
n_resamples):
#sample 1
adjusted_n_1 = (len_input_array_1 - 1)/(1 - sam_pop_ratio_1)
##some considerations for having a decimal adjusted_n_1
base_adjusted_n_1 = int(adjusted_n_1)
fraction_adjusted_n_1 = adjusted_n_1 - base_adjusted_n_1
#create an a array of sample 1 resample sizes
##alternate size to account for the fraction of adjustment
adjusted_n_array_1 = [base_adjusted_n_1 + \
int(np.random.choice([0,1], size = 1, \
p = [1 - fraction_adjusted_n_1, fraction_adjusted_n_1)) \
for x in range(n_samples)]
#sample 2 (same setup as above for sample 1)
adjusted_n_2 = (len_input_array_2 - 1)/(1 - sam_pop_ratio_2)
base_adjusted_n_2 = int(adjusted_n_2)
fraction_adjusted_n_2 = adjusted_n_1 - base_adjusted_n_2
adjusted_n_array_2 = [base_adjusted_n_2 + \
int(np.random.choice([0,1], size = 1, \
p = [1 - fraction_adjusted_n_2, fraction_adjusted_n_2)) \
for x in range(n_samples)]
pooled_array_n = np.add(adjusted_n_array_1, adjusted_n_array_2
pooled_array = input_array_1 + input_array_2
#create list of resampled medians for group 1 and group 2
medians_1 = [np.median(np.random.choice(pooled_array, size = x)) \
for x in adjusted_n_array_1]
medians_2 = [np.median(np.random.choice(pooled_array, size = x)) \
for x in adjusted_n_array_2]
n_resamples = 10000
bs_pool_delta = bootstrap_medians_pooled_approach(responses_1, len(responses_1),
sam_pop_ratio_1,\
responses_2, len(responses_2), sam_pop_ratio_2, \
n_resamples)
#visualize the distribution of deltas results
fig_bsed_pool_deltas = go.Figure()
fig_bsed_pool_deltas.add_trace(go.Histogram(x = bs_pool_delta)
#explore the chance that the observed delta of a given delta might be observed by random chance
deltas = 0.25 * x for x in range(-28,28)
fig_ps_bs = go.Figure()
fig_ps_bs.add_trace(go.Scatter(x = deltas, y = bsed_p_values_pool))
If I wasn't adjusting the group sizes for the finite population, I would shuffle the pooled data into new groups (without replacement). By resampling with replacement, how should I interpret the results? Is it still correct to think about the fig_bsed_pool_deltas
as the probability of observing the delta due to random error? Or is this a misapplication of the technique? One thing that bothers me is that I pool the data, but then use the original group size rather than setting the populations of each group to the sum of population_size_1 and population_size_2.
Note: If I don't pool all results but rather 1) resample median values for group 1, 2) resample median values for group 2, and then 3) find the resulting difference vector, I get a much tighter distribution of the delta. I think that I would interpret this as the distribution the the calculated median, not the probability of getting the result by random chance.