0
$\begingroup$

This might be considered a duplicate, but I don't have the ability to ask follow up questions in comments to other questions/answers yet.

For the χ2 statistic, we know that if all counts in the sample are scaled by the same factor, the calculated statistic will also scale by that factor, potentially reversing the outcome of a test. This makes me feel like χ2 tests are inconsistent.

I looked at Normalization the data before applying statistical test for large sample size, and I do realize that the same relative difference in frequencies at a larger sample size is more indicative of a mismatch between distributions.

I'm also aware that χ2 tests have a limitation on the minimum sample size. This makes them inappropriate in particular for testing with relative category frequencies rather than actual counts.

Now I'm wondering if there are additional assumption on the populations or samples when one could still perform, say, the χ2 test of homogeneity when only relative frequencies are available. E.g,

Preferred brand |  A   |  B   |  C
---
All of the US   | 0.45 | 0.3  | 0.25
Orange County   | 0.44 | 0.29 | 0.27

This is an overly simplified example, but I feel like it stands to reason that if the relative frequencies are very similar, like above, than the samples likely come from very close distributions. Where's a flaw in this logic?
Note: the two populations in the above example are purposefully of very different sizes, I'm especially interested in this case.

Also, could we use the known population sizes (e.g, the census data) to infer the counts, and then use the χ2 test even though the relative frequencies were naturally not based on the entire populations?

Would another test be more appropriate for testing the distribution likeness in this case?

$\endgroup$
6
  • 2
    $\begingroup$ You say: "This makes me feel like χ2 tests are inconsistent." .... Let me give you a simple framework. If I toss a coin twice and get 100% heads (i.e. heads twice), should I consider my coin biased toward heads? Or should I note that the results "all heads or all tails" will occur half the time with a fair coin and so not get excited? Now, my friend also tosses a coin and gets 100% heads -- but she tossed her coin 50 times. Even though we both got 100% heads with our individual coins, should our conclusions really be the same? You're saying it would be inconsistent if they were different. $\endgroup$
    – Glen_b
    Jun 18, 2020 at 4:11
  • $\begingroup$ That's a bit of an extreme example. But if I toss a coin 10 times and get 6:4 and then 100 times and get 60:40, that's not as unlikely to be from the same distribution, is it? $\endgroup$
    – Arseny
    Jun 18, 2020 at 4:21
  • 2
    $\begingroup$ The extreme example was to make it obvious but the point holds more generally; I advise you not to brush it off. With 10 tosses the chance that you get a result at least as far from fair as 6:4 is over 75% (i.e. it happens all the time) while with 100 tosses that ratio happens 5.7% of the time; and with 200 tosses it happens well under 1% of the time. Is it really inconsistent to make a different decision for some of those? $\endgroup$
    – Glen_b
    Jun 18, 2020 at 4:34
  • 2
    $\begingroup$ The issue is that you really can't just treat it in terms of the proportions. How surprising an observed proportion is will be intimately tied to the sample size. $\endgroup$
    – Glen_b
    Jun 18, 2020 at 4:36
  • 1
    $\begingroup$ Thanks! I think that's fair. I guess I'll have to dig up the sample sizes after all $\endgroup$
    – Arseny
    Jun 18, 2020 at 4:42

1 Answer 1

1
$\begingroup$

A sample count proportion is not equal to the population proportion.

Count proportions (e.g. from a binomial or hypergeometric) tend to vary more when they're close to the middle (a broad range of proportions near 0.5), less when they're close to zero or one.

A count proportion is a kind of average, and like other averages, they tend to be more variable when sample sizes are small and less variable when sample sizes are large (the standard error of a mean is $\sigma/\sqrt{n}$ here as well, but $\sigma$ is related to the true population proportion).

This means that a large difference in two sample proportions can very easily occur when the population proportions are equal if sample sizes are small, but only relatively small differences will occur when sample sizes are large. This means that sample size is critical to deciding whether a certain amount of change in proportion is indicating the population proportions differ or if it could be consistent with there being no difference in population proportions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.