# How do I estimate effective sample size from a bootstrapped sample?

Before now, I have typically used Kish's effective sample size estimator to estimate the precision in survey data with sample weights. I understand bootstrapping is another approach for estimating effective sample size, and I would like to compare the performance of the two in my data, especially for rare-events data. I have written a simple program to bootstrap counts from a sample with survey weights, with the chance of including a data point based on the survey weight. (These are later divided by the sample size to produce proportions.)

However, my guess at how to estimate the (effective) sample size from these bootstrapped estimates produces wildly divergent and nonsensical results, so I know I am doing something wrong.

Resampling from the data 20 times with weights, as a small example, I get the following values:

• Produces the counts: [4570, 4637, 4630, 4533, 4651, 4621, 4601, 4636, 4472, 4622, 4575, 4573, 4564, 4627, 4640, 4713, 4496, 4601, 4633, 4549]. Sample mean: 4597.2, sample variance: 3048.16.
• The original and weighted sample size of the data is 11937.
• This results in the proportions: [0.38284326, 0.38845606, 0.38786965, 0.37974365, 0.38962888, 0.38711569, 0.38544023, 0.38837229, 0.37463349, 0.38719946, 0.38326213, 0.38309458, 0.38234062, 0.38761833, 0.38870738, 0.39482282, 0.37664405, 0.38544023, 0.38812097, 0.38108402]. Sample mean: 0.3851, sample variance: $$2.1392\times10^{-5}$$.
• These count and proportion bootstrap sample means look to "dance" around the weighted count and proportion value in our dataset as expected, which are 4610.0 and 0.38620, respectively.

My intuition is that I should be able to estimate an effective sample size based on the variation observed in the bootstrapped sample. My idea is to use the variance of the bootstrapped proportions and the mean of the bootstrapped proportions to estimate the "effective" sample size, from the equation for the variance of a proportion, which as I understand is $$\sigma_p^2 = ( p * (1-p) ) / n$$.

• Rearranged and rewritten for our estimators, this would be: $$n_{eff} = ( p * (1-p) ) / s_p^2 = 0.3851 * (1-0.3851) / 2.1392\times10^{-5} =~ 11069.70$$
• However, the effective sample size according to Kish's estimator is 4509.60 (the weighting efficiency on this survey is very low).

What is the proper way to estimate this? Any sense of what I am doing wrong? I am still developing my intution around bootstrapping, so I suspect there is either a key logical step or a simple algebraic mistake I am making.

## Definition of Effective Sample Size

For a given statistic, the "effective sample size" is the sample size of a simple random sample that would yield the same sampling variance as the actual complex sample design. Equivalently, the effective sample size is the actual sample size divided by the "design effect" of the complex sample design.

In notation, suppose $$T$$ is some population quantity of interest for which you have an estimator, $$\hat{T}$$. Given the sample design for your data (which might include features such as stratification, clustering, unequal probabilities of selection), your estimator $$\hat{T}$$ has a sampling variance, denoted $$V(\hat{T})$$. This sampling variance is the expected variance of your estimator $$\hat{T}$$ across repeated sampling according to your complex design.

Hypothetically, you could have used a simple random sample instead of a complex sampling design. The variance of your estimator under simple random sampling is denoted $$V_{srs}(\hat{T})$$.

The term "design effect" refers to the ratio of the two variances, $$deff(\hat{T})=V(\hat{T}) / V_{srs}(\hat{T})$$.

The effective sample is the actual sample size divided by the design effect: $$n_{eff}(\hat{T}) = n /deff(\hat{T})$$.

## Estimating the Effective Sample Size

Generally speaking, we have to estimate $$V(\hat{T})$$ and $$V_{srs}(\hat{T})$$. The way you do this depends on the statistic you're estimating. For basic statistics like a mean or proportion, it's easy to estimate $$V_{srs}$$.

For a proportion, for example, we first estimate the population proportion $$p$$, and then we have $$V_{srs}(\hat{p})=n^{-1}\hat{p}(1-\hat{p})$$. And so $$n_{eff}(\hat{p})=\hat{p}(1-\hat{p})/V(\hat{p})$$. So for a proportion, the hard part is just to estimate $$V(\hat{p})$$.

In general, it's the sampling variance under the complex design, $$V(\hat{T})$$ that's harder to estimate. Having the sampling weights isn't enough except in certain special cases: see this StackExchange question. If your data come from something like the Census Bureau, they should have documentation explaining how to estimate the variances.

The bootstrap is one method for estimating variances, but it has to be adapted suitably for the complex survey designs. For example, if you have a stratified sample, then you need to bootstrap separately by stratum.

## Some recommended references on variance estimation for complex surveys

Books:

• Sharon Lohr’s textbook “Sampling: Design and Analysis”. Second or Third Editions.
• Thomas Lumley’s book “Complex Surveys: a guide to analysis using R”

Open-access articles:

Software:

• The 'survey' package in R, and the related package 'srvyr'. These make it easy to estimate variances and design effects for complex surveys.
• Thanks @bschneidr. Accepting this as an answer because it’s thorough and gives directions for future exploration. I suppose what confuses me is that Kish’s effective sample size estimator does use just the sampling weights. So is there a way to “thread the needle” with bootstrapping that may not be perfectly accurate but is an improvement on Kish’s estimator. Commented Nov 10, 2022 at 4:32
• Another question, reading the linked response: if I were to reweight following each bootstrap iteration, would this not work? Clustering is not a concern with my data and, as I understand it, stratification lowers variance in most cases, so omitting this is not a major concern. Commented Nov 10, 2022 at 4:48
• Could you explain how your sample was selected and how the weights were created? For complex surveys, there are several varieties of bootstraps that have been developed, some of which might be applicable to your data. Commented Nov 10, 2022 at 14:20
• the sample was stratified along demographic variables (classic ones like gender, ethnicity), then weighted along these and one other calibration variable to match the sample totals to the population totals. (My understanding is these are post-stratification weights / that these are inverse probabilities of selection.) Commented Nov 10, 2022 at 17:21