Derivation of the Simpson index I'm interested in the derivation of the Simpson diversity index. I found the normal interpretation of this index, but I'm not able to see what's the "mathematical background" of the formula. I read the original paper of Simpson, but I'm more interested in an easier explanation (my skills in probability theory are more on an undergraduate level). I would appreciate a little help here.
 A: The "normal" interpretation of the index is that it is an index to measure biodiversity.
Mathematically, the formula is $ D=\sum(n/N)^2$ or  $D=\sum\frac {n(n-1)} {N(N-1)}$
$n$ is the number of a given species and $N$ is the total population. What is being calculated is probability that two randomly selected individuals from a sample belong to the same species. 
The bigger the value of D, the more likely they are to be from the same species, hence the  lower the diversity. To fix this problem of counter-intuitiveness $1-D$ is calculated instead.
A: There are several versions of the Simpson diversity index, as explained in this website.  I will focus here on this version, which I have stated slightly more precisely to clarify what the sum is over:
$$D = \sum_{i=1}^k\frac{n_i(n_i-1)}{N(N-1)}$$
Here $N$ is the total number of individuals within a habitat, and $n_i$ is the number of individuals of the $i$th of $k$ species. Note that, since $N$ is not indexed by $i$, it makes no difference whether the denominator is within the scope of the sum.  We can equally write:
$$D =\frac{\sum_{i=1}^kn_i(n_i-1)}{N(N-1)}$$
The derivation of these formulae is a straightforward application of probability. If two individuals are chosen at random from the habitat, the total number of possible outcomes $O_{tot}$ is:
$$O_{tot} = N(N-1)/2$$
The division by $2$ is to avoid duplication where the same two individuals are chosen in reverse order.  The outcomes of interest $O_{int}$ are those in which the chosen two individuals belong to the same species. For any one species $i$ the number of pairs $P_i$ of individuals belonging to that species is:
$$P_i = n_i(n_i-1)/2$$
To find $O_{int}$ we must sum over all species:
$$O_{int} = \sum_{i=1}^kP_i = \sum_{i=1}^kn_i(n_i-1)/2$$
Dividing $O_{int}$ by $O_{tot}$ and cancelling the divisions by $2$ yields the second of the above formulae for $D$.
A: One way to derive the Simpson index is via the concept of surprise and expected surprise.  For definitions and background, see the posts Statistical interpretation of Maximum Entropy Distribution and How to include the observed values, not just their probabilities, in information entropy?. I will use the theoretical definition of the Simpson index as $\sum_1^n p_i²$. In a biodiversity setting, say, $i$ indices $n$ species, where the probability of observing species $i$ is $p_i$.
Now let the observable, the random variable, be $I$, the species index of the observed species. Then the surprise function of the observed species is
$$ S(I) = p(I)=p_i $$
the probability of observing that species, and the expected surprise is
$$ \DeclareMathOperator{\E}{\mathbb{E}}
   \E S(I)=\sum_i p_i S(i) =\sum_i p_i^2
$$  that is, it is the Simpson index!
