Shannon Index normalization with S distinct species per site

Question

Given S different species and M sites . Where in every site species are unique (not repeated within the site,that is there are no 2 individuals from the same species in the same site , but can be repeated across sites) . N random sites picked as a sample , and species occurrence rate counted in every site in the sample. Applying Shannon index across sites gives me different numbers, I am unable to find an upper bound for the index .
How can I normalize Shannon Index in this case ,or at least find the upper bond for the index ?

Answer 1

Define Shannon entropy as $H = \sum_{i=1}^S p_i \ln (1/p_i)$ where $p_i$ is the fractional abundance of species $i$, $i = 1, \dots, S$. Here 'species' may, but need not, mean biological species; any other kind of taxon or category may fit the bill. The rules are just that the 'species' are mutually exclusive and collectively exhaustive of a set of observations.

Using natural logarithms here is convenient and personally congenial: nothing rules out logarithms to other bases as a matter of tradition or taste, but some details below will change correspondingly. Many people want to use the rules of logarithms to rewrite that definition first as $H = \sum_{i=1}^S p_i (-\ln p_i)$ and then as $H = -\sum_{i=1}^S p_i \ln p_i$, and algebraically all these forms are equivalent. Similarly, in some circles or some circumstances spelling out the subscripts may seem merely cosmetic. I like the first form as allowing a verbal translation of entropy as the weighted average of $\ln (1/p)$, the problem being then to explain why that is the focus of interest, which I leave to other stories, as this one is already longer than intended.

The maximum attainable value for entropy is when all probabilities are equal at $p_i = 1/S$ so that $H$ is then the sum of $S$ terms all $(1/S) \ln [1/(1/S)]$, which emerges cleanly from the wash as $\ln S$. An arm-waving argument that will satisfy only some is that $H$ is a measure of uncertainty say about which species will be observed next, and is so necessarily maximal when the probabilities are all equal. Slightly more convincing is to play with numerical examples: First calculate $\ln S$ for $p_i$ equal at $1/S$. Then try to increase $H$ by increasing one $p_i$ away from $1/S$ and necessarily decreasing at least one other to preserve the sum of $p_i$ as $1$. It will be found that $H$ falls off quickly in any direction. Most rigorous of all will be a formal proof, which I happily leave to others.

The minimum of $H$ is when one probability is $1$ and the others $0$, in which case $H$ reduces to $1 \ln (1/1) = 0$. The implicit rule that for any $p = 0$ the product $p \ln (1/p) = 0$ may cause surprise for anyone recalling that $1/0$ and thus $\ln (1/0)$ is indeterminate, but it follows from a variety of arguments, the easiest of which (although a little arm-waving) is to contemplate the graph of $p \ln (1/p)$ as a function of $p$. However, a practical detail that can bite hard is that you may need to insist that your software follows this rule, as many programs will return some flavour of missing or not a number otherwise. Alternatively, one hallmark of competent code is that it doesn't fall over if any of the probabilities are $0$.

So $H$ varies between $0$ and $\ln S$ and if desired $H/\ln S$ varies between 0 and 1, and all these limits are attainable. If using logarithms to base 2 or 10 or any other base, the division necessarily must use the logarithm to that base.

The biology of how many individuals you have -- or even whether it makes sense to think about individuals at all -- consider corals, mosses, and so forth -- is immaterial here. All that bites are the proportions in different 'species'. Similarly the ecology of whether you are characterising one site or pooling data from several sites is interesting between friends, but otherwise does not perturb the main principle.

Note. Scaling entropy in this way seems to me to be a way to throw away information in the false hope of comparability. It follows from the maximum that $\exp(H)$ is a "numbers equivalent", roughly an approximation to the "equivalent number of equally common species", which will be typically be less than $S$. Although it's not the question, there is a numbers equivalent for the Gini-Simpson measure too, and comparing those two with the actual number of species $S$ is much more informative than scaling $H$ to fall in $[0, 1]$.