I want to get results about population A. But all I can study is subpopulation C of B, where B is a subpopulation of A (that is, C $\subset$ B $\subset$ A). Each member of A is classified into one of the groups G1, G2, G3 and G4. Selection from A into B is not random, but selection from B into C is random, although some members of B have better chances to be selected into C.

To test if subpopulation C is representative of A, I constructed the "contingency" table ...

      A             B          C        Total?
G1     9817678      621718     90196    10,529,592
G2     1280567      191298     30799    1,502,664
G3     426856       79707      11000    517,563
G4     9817678      701425     87996    10,607,099
Total  21,342,779   1,594,148  219,991  23,156,918

... ignoring my doubts about column "Total?". For example, grand total 23,156,918 is pretty meaningless, since there are only 21,342,779 physically distinct members.

And, of course, for such huge populations the chi-squares test insists that B is not representative for A, and C is not representative for B.

 d <- data.frame(
     A = c(9817678, 1280567, 426856, 9817678),
     B = c(621718, 191298, 79707, 701425),
     C = c(90196, 30799, 11000, 87996)),
   row.names = c('G1', 'G2', 'G3', 'G4'))

#   Pearson's Chi-squared test
#data:  d[, 1:2]
#X-squared = 160929.2, df = 3, p-value < 2.2e-16

#   Pearson's Chi-squared test
#data:  d[, 2:3]
#X-squared = 1539.55, df = 3, p-value < 2.2e-16

Am I applying the wrong statistical test here?

Now, if I would randomly sample 100 members from A, B and C, and get the "average" numbers for G1 - G4, the chi-squared will be pretty confident that C is representative of B:

 d.r100 <- sapply(d, function(x) round(100*x/sum(x)))
#      A  B  C
#[1,] 46 39 41
#[2,]  6 12 14
#[3,]  2  5  5
#[4,] 46 44 40

#   Pearson's Chi-squared test
#data:  d.r100[, 2:3]
#X-squared = 0.3943, df = 3, p-value = 0.9414

It feels wrong that the more data I process, the worse my representativeness becomes.

After all, if studying "subpopulation C" will fetch me the answer for "population A" plus-minus mere 2-3%, it will be fine. But how do I estimate my confidence intervals for "A" if all I can study is "C"?

  • $\begingroup$ Daniil, I've made some slight edits to correct LATEX and punctuation. Please double-check that this accurately reflects your intentions. $\endgroup$ – Sycorax May 24 '13 at 15:13

1) If you actually have the population, you don't use statistical tests as if the population were a sample. Any notion of random sampling is nonsense; the p-values are meaningless.

2) Even if what you say is the population were actually a sample, you don't test subsamples against the whole sample, but against the remainder; generally such tests rely on independence, which you can't have if the same observations are in both groups

3) Even if the first two issues weren't already making a nonsense of it, with such huge sample sizes, there's no point in hypothesis tests; you already know the answer - subgroups simply don't have exactly the same distribution as the rest, and with a large enough sample, you can identify trivial differences.

  • $\begingroup$ Well, I have finally figured the way to deal with my data - to use standard GSEA (Gene Set Enrichment Analysis). $\endgroup$ – Daniil Sep 13 '13 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.