# Sampling variance is reduced when removing duplicates — why?

I have carried out a simulation experiment where $n$ out of $N$ items $x_i$ were drawn with replacement with unequal probabilities $z_i$ from a finite population. (If it matters: $z_i = 1/(Nx_i)$ in my case, but perhaps there are more general results.)

Then, I estimated the population totals of the indicators $$y_i := \begin{cases}1: y_i = m \\ 0: \text{otherwise}\end{cases}$$ for each $m$ in two ways:

1. By applying the Hansen-Hurwitz estimator: $\hat{Y}_I = n^{-1} \sum_i (y_i/z_i)$.
2. By first removing the duplicates from the sample and then applying the Horvitz-Thompson estimator for the selection probabilites $\pi_i = 1 - (1 - n/N)^{y_i}$ (which is the probability of selecting the item $i$ at least once, i.e. one minus the probability of not selecting it at all, approximable by $x_i$ Bernoulli trials with success probability $1 - n/N$ each if $x_i \ll n$): $\hat{Y}_{II} = \sum_i(y_i/\pi_i)$.

This was repeated 1000 times.

I compared the estimates with the true values and observed the ratio between estimate and true value. I can clearly see from my experiments that the second estimate is better in terms of sampling variance. How to support this with theoretical results?

From Cochran (1977), Chapter 9, I see the variance of the HH estimator as $$V(\hat{Y}_I) = n^{-1}\sum_i^N z_i (y_i/z_i - Y)^2$$ and that of the HT estimator as $$V(\hat{Y}_{II}) = \sum_i^N \frac{1-\pi_i}{\pi_i}y_i^2 + 2\sum_i^N\sum_{j>i}^N \frac{\pi_{ij}-\pi_i\pi_j}{\pi_i\pi_j}y_iy_j$$

but I have trouble applying this to my case. First, does it mean that for HH the variance depends on $Y$ and for HT it doesn't? Second, I would assume $\pi_{ij} = \pi_i\pi_j$ since the inclusion of items is independent, and then the second term of the HT variance vanishes -- or am I wrong?

I appreciate any help.

## 1 Answer

I don't understand the motivation for removing duplicates and using the HT estimator on the particular probabilities you are using.

More appropriate is to accept that the drawing is with replacement and hence there are duplicates (why would this be a problem? - normally it makes things simpler); and use the correct probabilities $z_i$ in the HT estimator.

Edit:

A couple of other thoughts

1. You also probably need to apply a finite population correction factor.

2. By removing the duplicates you are in effect just moving to a smaller sampling size and changing from sampling with replacement to sampling without replacement.

Edit by StasK:

Continuing with point 2: what you get is sometimes called Poisson sample. I am pretty sure that the pairwise selection probabilities are wrong, and $\pi_{ij} \neq \pi_i \pi_j$ in this case. Generally, sampling without replacement, and with unequal probabilities, is a HUGELY complicated thing. Brewer and Hanif (1982) list about 50 methods to do it properly, although only for about a dozen of these methods the pairwise probabilities of selection are tractable.

• Thank you. My motivation here is that I would like to estimate the "true value" with the least possible error (on average), and to me the variance seems to be a good indicator here. I have two options, and the second, though it seems inappropriate to you, obviously (?) produces slightly better results. And I'm asking why. Also, the direct computation of the sampling variance gives me headaches, see the second part of my question. – krlmlr May 1 '12 at 19:18
• What do you mean by "better"? do you mean that the calculated variance of the estimator matches what you see when you repeat it many times, or that it is smaller? I'll also add a bit to my answer. – Peter Ellis May 1 '12 at 19:43
• On point 1: no, you don't need an fpc. FPC is an artifact of the SRS sampling, and is the special case of the Horvitz-Thompson estimator; in other words, it is a algebraic simplification of these pairwise selection probabilities that this estimator takes into account. That's a very subtle, and rarely understood, point. – StasK May 1 '12 at 20:06
• @PeterEllis: "Better" is determined by what I see in the experiments. I repeat the sampling 1000 times and compute the mean and variance of $q = (\text{estimated}-\text{true})/\text{true}$. In the WOR case, the variance is smaller, but for both estimators the mean is zero (this means they're unbiased, right?). I didn't try yet to apply the formulas to compute the variances, because I first wanted to get feedback if these are the correct formulas in the first place and why they look/seem so different. – krlmlr May 1 '12 at 20:09
• @StasK: Thank you for the hints, and also for your earlier recommendation of the Cochran book. I have found this earlier article from the keywords you provided: projecteuclid.org/… . The abstract effectively promises that the paper proves what I see in my experiment, but the maths involved are way too difficult for me. Any thoughts? – krlmlr May 1 '12 at 20:28