0
$\begingroup$

Lets say that:

There exist 10 countries.

I travel to each country and take a random subset of the population of that country and count how many of my subset are woman.

I make a table of country to percentage female for the 10 countries.

I then repeat this study, but instead of traveling to the countries, I use some other method of getting a supposedly random sample of people to check for femaleness.

In the end my data looks like this.

country, random-sampling, new-technique
a,       0.6,             0.55
b,       0.4,             0.38
c,       0.9,             0.85
.
.
.

If I assume that the first time around I actually found a representative sample of people for each country, and thus we trust that there really are 90% woman in country 'c' then:

How do I validate whether the new technique is a valid way of testing for percentage woman in countries? I want to know how well the second set of data agrees with the assumed-to-be-correct first set.

$\endgroup$
3
  • $\begingroup$ You'll want to assess whether your estimates are biased and have less variance over repeated sampling. But a question I have is do you consider the first method the "gold standard" that you'd like to compare to? $\endgroup$ Commented Apr 10, 2015 at 21:07
  • $\begingroup$ Yes, assuming that the first method is the 'gold standard'. $\endgroup$
    – avoid3d
    Commented Apr 10, 2015 at 21:13
  • $\begingroup$ To elaborate, I don't want to use the second-technique to add certainty some overall estimate, I want to validate whether the second-technique is a valid method, assuming that the first was 'perfect'. $\endgroup$
    – avoid3d
    Commented Apr 10, 2015 at 21:15

1 Answer 1

0
$\begingroup$

If you really want to take the first sample as the truth, then we can use the estimated probabilities from that sample as known probabilities, and use those to formulate a null hypothesis for the new-technique sample. The null hypothesis can be that the second sample is as if sampled with/without replacement (you didn't specify), lets assume without, so a hypergeometric model (with the given probability parameters $p_a=0.6, p_b=0.4, \dotsc$.) The sample sizes must also be given (you didn't) as $n_a, n_b, \dotsc$ and known. Then the data from the second sample is $X_a \sim \mathcal{Hypergeo}(n_a p_a,n_a(1-p_a), X_b \sim \dots$ (independently.) Just lets hope those products are integers ...

Then you can construct a hypergeometric test for each population, and somehow combine those to get an overall test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.