# What is the best way of measuring the dispersion or concentration of categorical data?

Let me provide some background. I am writing a paper on the effectiveness of specialist vs. generalist software companies. I have data on the number of products of theirs that are being used across several industries. I would like to be able to measure how concentrated the data is in a particular category (industry in this case) so I can more easily classify each company as a specialist or a generalist. Are there any standard or well known ways of doing this? I am developing my own metric at the moment but would like to know if there is a tried-and-true method of doing so.

Here is some example data:

Google: {Finance: 12000 products, Retail: 9000 products, Transport: 15000 products, Healthcare: 8000 products, Education: 11500 products ...}

Specialist Software Corporation: {Finance: 1500 products, Government: 90 products, Education: 68 products, ...}

Something to show that products are much more concentrated in one (or however many) industries for one company than for another (despite the other having a greater number of overall products).

• There is a large literature even within economics on this. Books by Henri Theil, Economics and Information Theory and its sequel, remain lucid, authoritative and relevant. – Nick Cox Nov 1 '17 at 21:58

For each company, you can quantify the dispersion using the Shannon Entropy: $H=-\sum_i p_i \mathrm{log}(p_i)$.
$p_i$ is the probability of product $i$ in your case. So for the Specialist Software Corporation, the probabilities are (1500/1658, 90/1658, 68/1658). Be sure to specify the base of the logarithm you use.
• Most people write $H$ like that. I think it's more easily understood on first acquaintance as $\sum_i p_i \log (1/p_i)$, i.e. as a weighted average of $\log (1/p_i)$. It's understandable -- as a matter of algebraic etiquette -- that people want to simplify first to $\sum_i p_i (-\log p_i)$ and then to the expression you give, but the starting point is $\log (1/p_i)$ as a measure of how much information there is in observing an event with probability $p_i$. Then comes the explanation of why that in the first place. – Nick Cox Nov 1 '17 at 21:11
• $\sum_i p_i^2$ works well too, as do its complement and its reciprocal. Associated names include Gini, Turing, Simpson, Hirschman, Herfindahl, and so on, and so on. Multiple reinventions and multiple names in multiple literatures. David MacKay and perhaps others, called this the match probability, a great name. – Nick Cox Nov 1 '17 at 21:14
• Consider the case of everything in one category. Then it's just the single value $-1 \log 1$, which is zero for any base of logarithms; so yes, closer to 0, the more concentrated. – Nick Cox Nov 1 '17 at 21:19