4
$\begingroup$

I'm trying to estimate the number of pairs that exist in my data (specifically, these are identical, or near-identical, images). Checking for pairs is expensive and time consuming.

My population size is known. In one case it is 1,000,000. I take a sample of 7500, and find 14 pairs.

Some simulations I've done suggest that:

$$ popPairs = (\frac{nPop}{nSample})^2 \times samplePairs $$

Where:

  • nPop = population size
  • nSample = sample size
  • samplePairs = number of pairs found in the population
  • popPairs = number of pairs in the population

What I would really like is a standard error on that estimate, but I'm not sure how to get one.

Edit: Colab with code can be found here: https://colab.research.google.com/drive/17LTijBGnEDBkl1slFVPw4VvxnXAJJlf7?authuser=1

$\endgroup$
2
  • $\begingroup$ Can you be a bit more specific about your sampling comparisons. For example, it is possible that some items exist in the population in quantities more than two (i.e., triples, quadruples, etc.). If so, does your sampling comparison detect triples, quadruples, etc.? Or would it detect a triple in the sample only as three pairs? $\endgroup$
    – Ben
    Apr 17, 2020 at 9:01
  • $\begingroup$ The detection would detect triples as three pairs. $\endgroup$ Apr 17, 2020 at 16:46

1 Answer 1

2
$\begingroup$

I'm really curious about the estimator you are using. This estimator validity is not really clear to me. For example if you take $nPop = 100$, $nSample = 10$ and $samplePairs = 2$ your would have $popPairs = 200 > nPop$.

I'm going to propose a slightly different approach.

Consider the following random variable $X$. You sample without repetition a pair of images, then we have $x = 1$ if the images are equal and $X = 0$ otherwise.

Now we have that $X$ is Bernoulli with mean $\mu =p / (n(n-1))$. ($n$ is population size and $p$ number of pairs.)

You can estimate $\mu$ by independently sampling pairs from your population and computing the average of observed $X$ values. Let's suppose we take $m$ random samples and compute $\hat \mu = \sum X_i / m$. then we have $E(\hat \mu) = \mu$ and $Var(X) = \mu (1-\mu) / m$.

Finally we take $Y = n(n - 1)\hat \mu$ then $E(Y) = p$ and $Var(Y) = \frac{np}{m}(n - 1 - p/n)$.

As your can see $Y$ is a random variable with mean equal to the quantity you want. However the variance can be quite high depending on $m$.

$\endgroup$
5
  • $\begingroup$ (1) Thanks for that. I've only tried with very high numbers - I don't know if it would work with smaller numbers. 2 pairs in a sample of 10 would (I think be unlikely, I got 2 or 3 pairs 6 times out of 100 in a simulation when the data comprised of 50 pairs, and the mean was 0.55, which is about right). $\endgroup$ Apr 17, 2020 at 4:39
  • $\begingroup$ (2) I don't think I can sample pairs. At least, I don't have that data. $\endgroup$ Apr 17, 2020 at 4:40
  • $\begingroup$ Your surely can sample pairs. Just sample one image and than sample another but now removing the image you just sample from the population $\endgroup$
    – Mur1lo
    Apr 17, 2020 at 4:51
  • $\begingroup$ :) You'd think so, but I'm not doing the sampling (and I wasn't asked before it was done, of course). $\endgroup$ Apr 17, 2020 at 5:28
  • 1
    $\begingroup$ That changes things :). Using your approach i would estimate using popPairs = samplePairs * nPop / nSample $\endgroup$
    – Mur1lo
    Apr 17, 2020 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.