Sampling variance is reduced when removing duplicates -- why?

Question

I have carried out a simulation experiment where $n$ out of $N$ items $x_i$ were drawn with replacement with unequal probabilities $z_i$ from a finite population. (If it matters: $z_i = 1/(Nx_i)$ in my case, but perhaps there are more general results.)

Then, I estimated the population totals of the indicators $$ y_i := \begin{cases}1: y_i = m \\ 0: \text{otherwise}\end{cases} $$ for each $m$ in two ways:

1. By applying the Hansen-Hurwitz estimator: $\hat{Y}_I = n^{-1} \sum_i (y_i/z_i)$.
2. By first removing the duplicates from the sample and then applying the Horvitz-Thompson estimator for the selection probabilites $\pi_i = 1 - (1 - n/N)^{y_i}$ (which is the probability of selecting the item $i$ at least once, i.e. one minus the probability of not selecting it at all, approximable by $x_i$ Bernoulli trials with success probability $1 - n/N$ each if $x_i \ll n$): $\hat{Y}_{II} = \sum_i(y_i/\pi_i)$.

This was repeated 1000 times.

I compared the estimates with the true values and observed the ratio between estimate and true value. I can clearly see from my experiments that the second estimate is better in terms of sampling variance. How to support this with theoretical results?

From Cochran (1977), Chapter 9, I see the variance of the HH estimator as $$ V(\hat{Y}_I) = n^{-1}\sum_i^N z_i (y_i/z_i - Y)^2 $$ and that of the HT estimator as $$ V(\hat{Y}_{II}) = \sum_i^N \frac{1-\pi_i}{\pi_i}y_i^2 + 2\sum_i^N\sum_{j>i}^N \frac{\pi_{ij}-\pi_i\pi_j}{\pi_i\pi_j}y_iy_j $$

but I have trouble applying this to my case. First, does it mean that for HH the variance depends on $Y$ and for HT it doesn't? Second, I would assume $\pi_{ij} = \pi_i\pi_j$ since the inclusion of items is independent, and then the second term of the HT variance vanishes -- or am I wrong?

I appreciate any help.

Answer 1

I don't understand the motivation for removing duplicates and using the HT estimator on the particular probabilities you are using.

More appropriate is to accept that the drawing is with replacement and hence there are duplicates (why would this be a problem? - normally it makes things simpler); and use the correct probabilities $z_i$ in the HT estimator.

Edit:

A couple of other thoughts

1. You also probably need to apply a finite population correction factor.

2. By removing the duplicates you are in effect just moving to a smaller sampling size and changing from sampling with replacement to sampling without replacement.

Edit by StasK:

Continuing with point 2: what you get is sometimes called Poisson sample. I am pretty sure that the pairwise selection probabilities are wrong, and $\pi_{ij} \neq \pi_i \pi_j$ in this case. Generally, sampling without replacement, and with unequal probabilities, is a HUGELY complicated thing. Brewer and Hanif (1982) list about 50 methods to do it properly, although only for about a dozen of these methods the pairwise probabilities of selection are tractable.