How can I quantify the amount of dispersion in a vector of word counts? I'm looking for a statistic that will be high for document A, because it contains many different words that occur infrequently, and low for document B, because it contains one word (or a few words) that occur often.
More generally, how does one measure the dispersion or "spread" in nominal data?
Is there a standard way of doing this in the text analysis community?
For probabilities (proportions or shares) $p_i$ summing to 1, the family $\sum p_i^a [\ln (1/p_i)]^b$ encapsulates several proposals for measures (indexes, coefficients, whatever) in this territory. Thus
$a = 0, b = 0$ returns the number of distinct words observed, which is the simplest to think about, regardless of its ignoring differences among the probabilities. This is always useful if only as context. In other fields, this could be the number of firms in a sector, the number of species observed at a site, and so forth. In general, let's call this the number of distinct items.
$a = 2, b = 0$ returns the Gini-Turing-Simpson-Herfindahl-Hirschman-Greenberg sum of squared probabilities, otherwise known as the repeat rate or purity or match probability or homozygosity. It is often reported as its complement or its reciprocal, sometimes then under other names, such as impurity or heterozygosity. In this context, it is the probability that two words selected randomly are the same, and its complement $1 - \sum p_i^2$ the probability that two words are different. The reciprocal $1 / \sum p_i^2$ has an interpretation as the equivalent number of equally common categories; this is sometimes called the numbers equivalent. Such an interpretation can be seen by noting that $k$ equally common categories (each probability thus $1/k$) imply $\sum p_i^2 = k (1/k)^2 = 1/k$ so that the reciprocal of the probability is just $k$. Picking a name is most likely to betray the field in which you work. Each field honours their own forebears, but I commend match probability as simple and most nearly self-defining.
$a = 1, b = 1$ returns Shannon entropy, often denoted $H$ and already signalled directly or indirectly in previous answers. The name entropy has stuck here, for a mix of excellent and not so good reasons, even occasionally physics envy. Note that $\exp(H)$ is the numbers equivalent for this measure, as seen by noting in similar style that $k$ equally common categories yield $H = \sum^k (1/k) \ln [1/(1/k)] = \ln k$, and hence $\exp(H) = \exp(\ln k)$ gives you back $k$. Entropy has many splendid properties; "information theory" is a good search term.
The formulation is found in I.J. Good. 1953. The population frequencies of species and the estimation of population parameters. Biometrika 40: 237-264. www.jstor.org/stable/2333344.
Other bases for logarithm (e.g. 10 or 2) are equally possible according to taste or precedent or convenience, with just simple variations implied for some formulas above.
Independent rediscoveries (or reinventions) of the second measure are manifold across several disciplines and the names above are far from a complete list.
Tying together common measures in a family is not just mildly appealing mathematically. It underlines that there is a choice of measure depending on the relative weights applied to scarce and common items, and so reduces any impression of adhockery created by a small profusion of apparently arbitrary proposals. The literature in some fields is weakened by papers and even books based on tenuous claims that some measure favoured by the author(s) is the best measure that everyone should be using.
My calculations indicate that examples A and B are not so different except on the first measure:
----------------------------------------------------------------------
| Shannon H exp(H) Simpson 1/Simpson #items
----------+-----------------------------------------------------------
A | 0.656 1.927 0.643 1.556 14
B | 0.684 1.981 0.630 1.588 9
----------------------------------------------------------------------
(Some may be interested to note that the Simpson named here (Edward Hugh Simpson, 1922-2019) is the same as that honoured by the name Simpson's paradox. He did excellent work, but he wasn't the first to discover either thing for which he is named, which in turn is Stigler's paradox, which in turn....)
I don't know if there's a common way of doing it, but this looks to me analogous to inequality questions in economics. If you treat each word as an individual and their count as comparable to income, you're interested in comparing where the bag of words is between the extremes of every word having the same count (complete equality), or one word having all the counts and everyone else zero. The complication being that the "zeros" don't show up, you can't have less than a count of 1 in a bag of words as usually defined ...
The Gini coefficient of A is 0.18, and of B is 0.43, which shows that A is more "equal" than B.
library(ineq)
A <- c(3, 2, 2, rep(1, 11))
B <- c(9, 2, rep(1, 7))
Gini(A)
Gini(B)
I'm interested in any other answers too. Obviously the old fashioned variance in counts would be a starting point too, but you'd have to scale it somehow to make it comparable for bags of different sizes and hence different mean counts per word.
This article has a review of standard dispersion measures used by linguists. They are listed as single-word dispersion measures (They measure the dispersion of words across sections, pages etc.) but could conceivably be used as word frequency dispersion measures. The standard statistical ones seem to be:
The classics are:
Where $N$ is the total number of words in the text, $n$ is the number of distinct words, and $n_i$ the number of occurrences of the i-th word in the text.
The text also mentions two more measures of dispersion, but they rely on the spatial positioning of the words, so this is inapplicable to the bag of words model.
The first I would do is calculating Shannon's entropy. You can use the R package infotheo, function entropy(X, method="emp"). If you wrap natstobits(H) around it, you will get the entropy of this source in bits.
One possible measure of equality you could use is the scaled Shannon entropy. If you have a vector of proportions $\boldsymbol{p} \equiv (p_1, ... , p_n)$ then this measure is given by:
$$\bar{H}(\boldsymbol{p}) \equiv - \frac{\sum p_i \ln p_i}{\ln n}.$$
This is a scaled measure with range $0 \leqslant \bar{H}(\boldsymbol{p}) \leqslant 1$ with extreme values occurring at the extremes of equality or inequality. Shannon entropy is a measure of information, and the scaled version allows comparison between cases with different numbers of categories.
Extreme Inequality: All the count is in some category $k$. In this case we have $p_i = \mathbb{I}(i=k)$ and this gives us $\bar{H}(\boldsymbol{p}) = 0$.
Extreme Equality: All the counts are equal over all categories. In this case we have $p_i = 1/n$ and this gives us $\bar{H}(\boldsymbol{p}) = 1$.
