# How do I compare a new method of statistical sampling with an established one?

Lets say that:

There exist 10 countries.

I travel to each country and take a random subset of the population of that country and count how many of my subset are woman.

I make a table of country to percentage female for the 10 countries.

I then repeat this study, but instead of traveling to the countries, I use some other method of getting a supposedly random sample of people to check for femaleness.

In the end my data looks like this.

country, random-sampling, new-technique
a,       0.6,             0.55
b,       0.4,             0.38
c,       0.9,             0.85
.
.
.


If I assume that the first time around I actually found a representative sample of people for each country, and thus we trust that there really are 90% woman in country 'c' then:

How do I validate whether the new technique is a valid way of testing for percentage woman in countries? I want to know how well the second set of data agrees with the assumed-to-be-correct first set.

• You'll want to assess whether your estimates are biased and have less variance over repeated sampling. But a question I have is do you consider the first method the "gold standard" that you'd like to compare to? Commented Apr 10, 2015 at 21:07
• Yes, assuming that the first method is the 'gold standard'. Commented Apr 10, 2015 at 21:13
• To elaborate, I don't want to use the second-technique to add certainty some overall estimate, I want to validate whether the second-technique is a valid method, assuming that the first was 'perfect'. Commented Apr 10, 2015 at 21:15

If you really want to take the first sample as the truth, then we can use the estimated probabilities from that sample as known probabilities, and use those to formulate a null hypothesis for the new-technique sample. The null hypothesis can be that the second sample is as if sampled with/without replacement (you didn't specify), lets assume without, so a hypergeometric model (with the given probability parameters $$p_a=0.6, p_b=0.4, \dotsc$$.) The sample sizes must also be given (you didn't) as $$n_a, n_b, \dotsc$$ and known. Then the data from the second sample is $$X_a \sim \mathcal{Hypergeo}(n_a p_a,n_a(1-p_a), X_b \sim \dots$$ (independently.) Just lets hope those products are integers ...