Estimate number of pairs in population of known size from sample

Question

I'm trying to estimate the number of pairs that exist in my data (specifically, these are identical, or near-identical, images). Checking for pairs is expensive and time consuming.

My population size is known. In one case it is 1,000,000. I take a sample of 7500, and find 14 pairs.

Some simulations I've done suggest that:

$$ popPairs = (\frac{nPop}{nSample})^2 \times samplePairs $$

Where:

• nPop = population size
• nSample = sample size
• samplePairs = number of pairs found in the population
• popPairs = number of pairs in the population

What I would really like is a standard error on that estimate, but I'm not sure how to get one.

Edit: Colab with code can be found here: https://colab.research.google.com/drive/17LTijBGnEDBkl1slFVPw4VvxnXAJJlf7?authuser=1

Answer 1

I'm really curious about the estimator you are using. This estimator validity is not really clear to me. For example if you take $nPop = 100$, $nSample = 10$ and $samplePairs = 2$ your would have $popPairs = 200 > nPop$.

I'm going to propose a slightly different approach.

Consider the following random variable $X$. You sample without repetition a pair of images, then we have $x = 1$ if the images are equal and $X = 0$ otherwise.

Now we have that $X$ is Bernoulli with mean $\mu =p / (n(n-1))$. ($n$ is population size and $p$ number of pairs.)

You can estimate $\mu$ by independently sampling pairs from your population and computing the average of observed $X$ values. Let's suppose we take $m$ random samples and compute $\hat \mu = \sum X_i / m$. then we have $E(\hat \mu) = \mu$ and $Var(X) = \mu (1-\mu) / m$.

Finally we take $Y = n(n - 1)\hat \mu$ then $E(Y) = p$ and $Var(Y) = \frac{np}{m}(n - 1 - p/n)$.

As your can see $Y$ is a random variable with mean equal to the quantity you want. However the variance can be quite high depending on $m$.