7
$\begingroup$

The Herfindahl–Hirschman Index (HHI) is a concentration measure defined as $$H = \sum_i p_i^2,$$ where $p_i$ is the market share of firm $i$. It's maximized when one firm has a monopoly and minimized when all firms have equal market shares.

Similarly, the entropy of a stochastic variable is defined as $$H = -\sum_i p_i \log p_i,$$ where $p_i$ is the probability of value $i$. It is maximized when all values are equally likely, and minimized when only one state is possible.

It seems clear that these are extremely similar, with entropy measuring the inverse (diversity) of the HHI (concentration). Many sources I've read note that they are 'similar' without being more precise. Are there any qualitative differences between the two?

$\endgroup$
4
  • 1
    $\begingroup$ The Wikipedia entry looks good for economists; but even with some wider references it underplays the extent to which this has many, many names and a longer history than stated, but it is neither possible nor appropriate to try a fuller account here. $\endgroup$
    – Nick Cox
    Commented Apr 21, 2020 at 12:15
  • 1
    $\begingroup$ I'm a physicist though, so much more familiar with entropy. Still rusty enough to be confused over them seemingly measuring the same thing qualitatively, yet being different. The paper linked in Kristian's answer was quite helpful in explaining this. $\endgroup$ Commented Apr 21, 2020 at 15:25
  • $\begingroup$ Although early schooling and general practices make many people immediately realise that $\sum p \log(1/p) = \sum -p \log p$ and then $- \sum p \log p$, I find some pedagogical advantage in writing $\sum p \log(1 / p)$ which allows emphasis on entropy as a weighted average. $\endgroup$
    – Nick Cox
    Commented Apr 21, 2020 at 16:21
  • $\begingroup$ Surely one of the biggest differences will be additivity: the entropy for two independent things just sums. The HHI won’t have that property $\endgroup$
    – innisfree
    Commented Mar 19, 2021 at 4:36

4 Answers 4

4
+100
$\begingroup$

In biology, these are called measures of diversity, and while that application is different, there must be some value in the comparison. See for example this wiki or this book by Anne Magurran. In that application $p_i$ is population share (probability that an individual sampled from the population is of species $i$.) For a very different application What is the probability that a person will die on their birthday?.

Anne Magurran strongly advises use of the Simpson index. The reason is that it does not depend so strongly on the long tail of small $p_i$'s, while the Shannon index (entropy) depends more on this. For that reason the Shannon index depends in practice on sample size (to a stronger degree than the Simpson index). But that might not be important in your economic application. In biology there is the aspect of unsampled species, if you have a full census of firms that should not be a problem. One idea to aid interpretation, to put such indices on a similar footing, is converting them to an equivalent number of species, the number of species which, with all $p_i$'s equal, would give the observed index value. For your application this would be equivalent number of firms. With this interpretation there are the Hill numbers $$ H_a = \left( \sum_i p_i^a \right)^{\frac1{1-a}} $$ which gives Simpson for $a=2$ (transformed), Shannon index for $a=1$ and number of species for $a=0$. This again shows that Shannon is closer to the number of species than is Simpson, so depends to a stronger degree on the many small $p_i$'s. So, qualitatively, the Simpson index depends more on the larger firms, while the Shannon index has a stronger influence from the smaller ones.

$\endgroup$
3
  • 1
    $\begingroup$ Another general family is $\sum_i p_i^a [\ln (1/p_i)]^b$ so number of species is given by $a = b = 0$, HHI is $a = 2, b = 0$, Shannon entropy is $a = 1, b = 1$ (Good, Biometrika 1953). As others have flagged, HHI has been reinvented or rediscovered many times; among many other names, quadratic entropy is one I have seen. $\endgroup$
    – Nick Cox
    Commented Apr 21, 2020 at 8:43
  • 2
    $\begingroup$ The popularity of various measures in any field seems to be related to whether its students, or even its practitioners, are all expected to understand logarithms. I have to flag that Magurran's book, while well known in ecology, was very carelessly copy edited and proof-read. As a technical guide it is unreliable. As a broad-brush introduction with engagement and enthusiasm about the topic as an ecological concept it arouses some affection in its field. $\endgroup$
    – Nick Cox
    Commented Apr 21, 2020 at 16:14
  • 1
    $\begingroup$ (ctd) I suspect that many workers used it as an introduction early in their careers and then moved on to more specific papers, and either never noticed or don't care about its technical slips. $\endgroup$
    – Nick Cox
    Commented Apr 21, 2020 at 16:17
2
$\begingroup$

I believe many sources refer to them as similar simply because both functionals are often used towards the same goal - quantifying the diversity/information of a given probability distribution. The HHI index in fact has many other names in different scientific disciplines, most notably the Simpson index.

An extensive and very readeable qualitative discussion can be found both in the Wikipedia article linked above and this paper, among many other sources.

For what it's worth, one can get $$ HHI(p) \geq \exp(-H(p))$$ via (weighted) Jensen's as follows:

$$ \exp(-H(p)) = \exp\left(\sum_i p_i\log p_i\right) = \prod_i p_i^{p_i} \overset{Jensen's}{\leq} \sum_i p_i\cdot p_i = HHI(p).$$

$\endgroup$
2
$\begingroup$

A few comments. Let $P = (p_1, p_2, \ldots, p_N)$ be a probability distribution (so that $0 \le p_i \le 1$ and $\sum_i p_i = 1$).

  • The measures are conceptually very closely related. The entropy is the expected surprise of a random draw from the distribution $P$ (where the surprise of an event with probability $p$ is defined to be $-\log(p)$). The HHI is the expected probability of a random draw from the distribution $P$. Probability is sort-of inverse to surprise, since it measures how likely something is, as opposed to how surprising it is.

  • The HHI is also the probability that two different random samples from $P$ have the same value.

  • Both measures ignore zero probabilities ($p\log(p)$ is defined to be zero if $p=0$ by convention).

There is also a numerical relationship between them. Let $\overline{P} = \frac{1}{N-1}(1-p_1, \ldots, 1-p_N)$. This is a probability distribution, which you could call the complement of $P$. Using the fact that $\log(1-p_i) \approx -p_i$ for $p_i \approx 0$, you can obtain

$$H(\overline{P}) \approx \frac{-1}{N-1}HHI(P) + \log(N-1) + \frac{1}{N-1}$$

provided that all the $p_i$ are fairly small, and you take natural log. So, morally, up to addition and multiplication by positive scalars, $HHI$ is the negative of the entropy of the "complement" distribution, which in turn is a kind of negative of the original distribution.

There is a paper about this on the arXiv which you could look at to see how these ideas are pursued. However, I would take its grandiose claims about "discovering extropy" with a grain of salt, as at least one of the authors is known to be a bit of a crank!

I think the main qualitative difference between the two measures is that entropy is only defined up to a scalar, because it depends on a choice of base for the logarithm ($e$ and $2$ being common choices) whereas for $HHI$ there is a natural scaling factor of $1$.

It seems that HHI is a very reasonable thing to use as a measure of diversity. However, I can't shake the feeling that entropy is "better" in terms of theoretical properties (such as those listed in Wikipedia).

$\endgroup$
3
  • $\begingroup$ "The HHI is the expected probability of a random draw from the distribution" is not a definition. $\endgroup$
    – Nick Cox
    Commented Apr 21, 2020 at 11:01
  • $\begingroup$ Let $X$ follow the distribution $P$. Define $f:\{1,\ldots,n\} \rightarrow \mathbb{R}$ by $f(i) = p_i$. Then $E[f(X)] = \sum_i \mathrm{Pr}(X=i)f(i) = \sum_i p_i^2$. $\endgroup$
    – Flounderer
    Commented Apr 21, 2020 at 11:33
  • $\begingroup$ Thanks; I should rephrase: not a transparent definition without more detail. $\endgroup$
    – Nick Cox
    Commented Apr 21, 2020 at 12:12
1
$\begingroup$

The first thing to notice is that each of these measures is in opposite directions, and they are also on different scales. In order to compare them in the same direction and scale, I am going to compare scaled versions of the negated HHI and entropy. Specifically, I will begin by comparing the following functions:

$$\begin{aligned} R(\mathbf{p}) &\equiv \frac{n-1}{n} \bigg( 1 - \sum_{i=1}^n p_i^2 \bigg), \\[6pt] S(\mathbf{p}) &\equiv - \frac{1}{\log n} \sum_{i=1}^n p_i \log p_i. \\[6pt] \end{aligned}$$

The HHI and the entropy are affine transformations of these two functions, so if we compare these two scaled functions, we will get simple corresponding results for the measures of interest. To see why I have chosen to examine these two functions, consider the special input vectors $\mathbf{u} \equiv (\tfrac{1}{n},...,\tfrac{1}{n})$ (all probabilities equal) and $\mathbf{m} \equiv (1,0,...,0)$ (one probability dominating). At these extremes we have the following results:

$$\begin{matrix} R(\mathbf{m}) = 0 & & & & R(\mathbf{u}) = 1, \\[6pt] S(\mathbf{m}) = 0 & & & & S(\mathbf{u}) = 1. \\[6pt] \end{matrix}$$

You can see from the above that the scaled functions I am using range between zero and one; they attain the zero value when one probability dominates the others and they attain unity when all the probabilities are equal. This means that both functions $R$ and $S$ are scaled measures of equality.


Rates-of-change of scaled equality measures: From the above forms of the functions, hopefully you can get a sense of the difference in the scaled measures. Below we will show the rates-of-change of the measures for a change in the probability vector. We will show that increasing a given probability will increase or decrease $R$ depending on whether that probability is below or above the arithmetic mean of the other probabilities. Contrarily, increasing a given probability will increase or decrease $S$ depending on whether that probability is below or above the geometric mean of the other probabilities.

We will examine rates-of-change as we alter one of the probabilities, with corresponding changes in other probabilities. To retain the norming requirement for the probability vector, we will consider that increasing the probability $p_k$ by some small amount $d p$ is accompanied by a corresponding change in all the other probabilties of $- \tfrac{1}{n-1} d p$. Thus, we have:

$$\frac{d p_i}{d p_k} = - \frac{1}{n-1} \quad \quad \quad \text{for } i \neq k.$$

Using the chain rule for total derivatives, for any $\mathbb{p}$ in the interior of its allowable range we therefore have:

$$\begin{aligned} \frac{d R}{d p_k} (\mathbf{p}) &= \sum_{i=1}^n \frac{d p_i}{d p_k} \cdot \frac{\partial R}{\partial p_i} (\mathbf{p}) \\[6pt] &= \frac{\partial R}{\partial p_k} (\mathbf{p}) + \sum_{i \neq k} \frac{d p_i}{d p_k} \cdot \frac{\partial R}{\partial p_i} (\mathbf{p}) \\[6pt] &= - \frac{n-1}{n} \cdot 2 p_k + \sum_{i \neq k} \frac{1}{n-1} \cdot \frac{n-1}{n} \cdot 2 p_i \\[6pt] &= - 2 \cdot \frac{n-1}{n} \Bigg[ p_k - \frac{1}{n-1} \sum_{i \neq k} p_i \Bigg], \\[6pt] \end{aligned}$$

and:

$$\begin{aligned} \frac{d S}{d p_k} (\mathbf{p}) &= \sum_{i=1}^n \frac{d p_i}{d p_k} \cdot \frac{\partial S}{\partial p_i} (\mathbf{p}) \\[6pt] &= \frac{\partial S}{\partial p_k} (\mathbf{p}) + \sum_{i \neq k} \frac{d p_i}{d p_k} \cdot \frac{\partial S}{\partial p_i} (\mathbf{p}) \\[6pt] &= - \frac{1}{\log n} \Bigg[ (1 + \log p_k) - \frac{1}{n-1} \sum_{i \neq k} (1 + \log p_i) \Bigg] \\[6pt] &= - \frac{1}{\log n} \Bigg[ \log p_k - \frac{1}{n-1} \sum_{i \neq k} \log p_i \Bigg]. \\[6pt] \end{aligned}$$

We can see that the two measures have different "cross-over points" for when an increase to $p_k$ increases or decreases the measure. For the measure $R$ the cross-over point is where $p_k$ is equal to the arithmetic mean of the other probabilities; below this point, increasing $p_k$ increases the measured equality between the elements and so it increases $R$. For the measure $S$ the cross-over point is where $p_k$ is equal to the geometric mean of the other probabilities; below this point, increasing $p_k$ increases the measured equality between the elements and so it increases $R$.


Relative rates-of-change and limiting cases: Aside from having different "cross-over" points, the two measures also change at different rates relative to one another when we change $p_k$. For a small increase in the probability $p_k$ we have:

$$\frac{dR}{dS} (\mathbf{p}) = \frac{d R}{d p_k} (\mathbf{p}) \Bigg/ \frac{d S}{d p_k} (\mathbf{p}) = \frac{2 (n-1) \log n}{n} \cdot \frac{p_k - \frac{1}{n-1} \sum_{i \neq k} p_i}{\log p_k - \frac{1}{n-1} \sum_{i \neq k} \log p_i}.$$

It is useful to examing this relative rate-of-change in the extreme cases. In particular, we have:

$$\lim_{p_k \uparrow 1} \frac{dR}{dS} (\mathbf{p}) = 0 \quad \quad \quad \lim_{p_k \downarrow 0} \frac{dR}{dS} (\mathbf{p}) = 2 \cdot \frac{n-1}{n} \cdot \frac{\log n}{\sum_{i \neq k} \log p_i}.$$

This shows that when $p_k$ is a dominating probability, which is near one, increasing it further will decrease $S$ much more rapidly than it decreases $R$. Contrarily, when $p_k$ is a dominated probability, which is near zero, increasing it increases $S$ much more rapidly than it increases $R$, and this is especially pronounced when $n$ is large.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.