2
$\begingroup$

We want to compare two distributions of ages (birth years) of individuals. Given a set of individuals (all) and a subset of that set (subset), we want to find out:

  1. Is it valid to compare the age distribution of all with that of subset (when |subset| is much smaller than |all|) or can we only compare all minus subset?
  2. We did a Shapiro-Wilk test on both distributions and got W = 0.7456 (p-value = 7.499e-08) for all and W = 0.7467 (p-value = 7.865e-08) for subset. This confirms that both distributions are not normal distributions, rather exponential or power-law. Therefore, we can not use Pearson correlation to compare them but should rather use Spearman?
  3. Comparing both distributions with Spearman, R says

    t = 93.9151, df = 47, p-value < 2.2e-16
    alternative hypothesis: true correlation is not equal to 0 
    95 percent confidence interval:
     0.9952748 0.9985103 
    sample estimates:
          cor 
    0.9973462
    

    which shows a high correlation. How meaningful is this result, given the large amount of young people?

  4. Can we infer from the correlation test that both distributions are (very) similar?
  5. If so, does this allow us to conclude that subset is not more biased towards younger people than all is?
  6. Finally, if our assumption was that subset contains more younger people than one would expect, can we now infer that this assumption is wrong?

Since I can not upload images, yet: here is a plot of both distributions and here the raw data:

# year  age(all)  age(subset)
1938  368  1
1939  360  1
1941  394  1
1942  809  3
1943  964  1
1944  686  1
1945  701  1
1946  1301  1
1947  1228  5
1948  1565  2
1949  2019  1
1950  2146  3
1951  2202  6
1952  2343  5
1953  2061  7
1954  2313  3
1955  2963  11
1956  3157  8
1957  3676  16
1958  4051  12
1959  5024  18
1960  5282  19
1961  6849  29
1962  7376  21
1963  8951  38
1964  10314  29
1965  13052  60
1966  13601  65
1967  14606  68
1968  18571  97
1969  19796  101
1970  21248  101
1971  20997  101
1972  24852  135
1973  26648  145
1974  30310  170
1975  34124  190
1976  39344  232
1977  45367  240
1978  54883  303
1979  63037  302
1980  73844  390
1981  77437  377
1982  83985  428
1983  90484  482
1984  100153  500
1985  100139  526
1986  106011  529
1987  101197  472
$\endgroup$
1
$\begingroup$

Partial answers:

  1. Should compare subset to all-subset, but it's not going to make a lot of difference.

I'd use a Kolmogorov-Smirnoff test both to test the similarity of the distributions and to highlight where such differences may exist.

3,4: I'm not sure the correlation means a lot, other than the fact that recent years have more of both.

$\endgroup$
  • $\begingroup$ Thanks! I had my doubts, too, thus I somehow expected your answer to 3+4. The KS test also shows a high similarity but when I played a bit around I realized that 5+6 likely don't hold: In subset I increased the counts for the eighties by a constant amount (say 100 individuals) and still got the same high correlation. So we could have more young people without seing a difference. I guess saying that both distributions are similar and there is no obvious difference is correct, but there is no statistical evidence for that or more complex conclusions. $\endgroup$ – Robert Aug 23 '13 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.