# Variance (maybe?) of categorical data

I don't really know the correct terminology to ask this question well. I have categorical data with counts and I want a measure of how "diverse" or "spread out" the data is. Variance comes to mind, but I don't know if that applies here.

Here is a two example of three distributions:

Sample 1
A: 100
B: 0
C: 0

Sample 2
A: 20
B: 20
C: 20


Things to note: the overall size between samples will not be consistent (i.e. how sample 1 has 100 counts and sample 2 has 60 counts) but I would like to compare between samples. In addition sample 1 and sample 2 represent opposite ends of the spectrum of the metric I would like where sample 2 is uniformly distributed (highly variant) and sample 1 has minimal variance. In addition the data is categorical so (A:10, B:0, C:10) should be equivalent to (A:0, B:10, C:10) when evaluated.

• It might be simpler to address the updated information by posting a new question, leaving the prior version of the question here along with the answer from @Tim. That will also minimize confusion for future visitors to this page.
– EdM
Commented Aug 8, 2019 at 20:28
• @EdM Thank you for the advice I will do that Commented Aug 8, 2019 at 20:31
• There are lots of questions on site that deal with diversity/ concentration indices and related measure of categorical "spread". Start here: en.wikipedia.org/wiki/Diversity_index (also see en.wikipedia.org/wiki/Herfindahl%E2%80%93Hirschman_Index; it's effectively the Simpson Index but the article links to the same notion existing in multiple areas). Commented Aug 9, 2019 at 0:31
• @Glen_b thank you! Just at a quick glance that looks like it's going to point me in the right direction. Commented Aug 9, 2019 at 0:37
• Note that the "variance" of a set of proportions is closely related to the Simpson index. Commented Aug 9, 2019 at 0:39

You can always look at the variance of the counts, but looking at your description, entropy seems to be a natural choice, since it meets all of your criteria. Entropy is defined as

$$S = -\sum_i p_i \log p_i$$

where $$p_i$$ is a probability of observing $$i$$-th category. The more uniform would be the distribution, the higher entropy it displays, so it is about being "diverse" vs uniform.

In your case, you have counts, so you can use them to calculate the empirical probabilities

$$\hat p_i = \frac{n_i}{\sum_j n_j}$$

where $$n_i$$ is the count for the $$i$$-th category. Since you have exact zeros in the counts, you should use some estimator of the probabilities that "smoothes" the zeros, since otherwise the formula for the entropy would not work (single zero would zero-out everything), one approach could be using a Bayesian estimator like Laplace smoothing, i.e.

$$\hat p_i = \frac{n_i+\alpha}{\sum_j n_j+\alpha}$$

where $$\alpha$$ is some constant, e.g. $$\alpha=1$$. In R this translates to:

> prob <- function(n, alpha=1) (n+alpha)/sum(n+alpha)
> entropy <- function(p) -sum(p*log(p))
> entropy(prob(c(0, 0, 100)))
[1] 0.1092225
> entropy(prob(c(20, 20, 20)))
[1] 1.098612
> entropy(prob(c(10, 0, 10)))
[1] 0.8418553
> entropy(prob(c(0, 10, 10)))
[1] 0.8418553


As you can see, sample 1 has low entropy, while sample 2 has high entropy. For samples 3 and 4, the entropy is the same, higher then for sample 1 (they are less extreme), but lower then sample 2 that is uniformly distributed.

• Thanks for the quick response! While entropy would be a decent metric you're right I would have to implement some sort of smoothing. I addition I would have to normalize my counts as entropy of raw counts will not be comparable between samples. While neither of these are prohibitive to using Entropy as my metric I'm worried that I will make poor choices on how to normalize my counts and how to smooth the counts, as there are many options for both. (and I clearly am not a statistician) Commented Aug 8, 2019 at 19:56
• @jTables looking at your numbers, incrementing them by 1 would not seem to have big impact on their magnitude, nor on their relative differences, so should not be an issue if you only want to compare the distributions. Using $\alpha=1$ is equivalent to using uniform prior for Bayesian estimator.
– Tim
Commented Aug 8, 2019 at 20:00
• Unfortunately these are just toy examples. Between samples the sparsity varies greatly, and there are up too 160 categories. I'm going to edit my question to include a better summary of the problem as I think I over simplified it and there is more than I can communicate in a comment Commented Aug 8, 2019 at 20:13
• @jTables still, if you choose a single constant for all the data, all the estimates of entropy would be off by the same quantity. The smoothing would make the counts more uniform, the degree on how much would it impact the counts depends on the choice of $\alpha$. You can always choose a small value, since the only point of this in your case would be to get rid of zeros. Entropy is a natural choice of measure for this kind of problems, the only problem are the zeroes.
– Tim
Commented Aug 8, 2019 at 20:18
• That is absolutely true for the description I gave. I added an update and I don't think it be comparable within sample between loci as the N would be different for each locus (13-160). Thank you again for helping me out, I appreciate the time you are putting in! I'm going to accept your answer as it was correct for how I originally framed the question. Commented Aug 8, 2019 at 20:25