i have data on how a large population (
N ~ 1e8) is distributed into (many) categories (i.e. i have count of instances in each category).
some categories have low counts, many categories have a high number of counts. there is no meaningful ordering of the categories.
i also have information on how a particularly selected subset (
n ~ 10k) from this population is distributed into the same categories (though the subset has zero counts in some of the population's categories).
i want to test the research hypothesis that the subset has a different distribution into the categories from that of the population. my null hypothesis is that the subset is a uniform random sample from the given population.
should the null hypothesis be rejected, i would furthermore like to identify which of the categories are significantly under/over represented in the subset compared to the population.
to this end i have tried this:
- trimmed the set of categories under consideration to only include the categories realised by the subset.
- computed the category ranking of each set
- tried to fit the problem into a friedman test.
now, my questions to you are:
- what is the most appropriate test statistic for the given hypothesis?
- does the friedman test apply here?
- how would you find which categories are over or under populated assuming the distributions are found to differ?