Quantifying diversity of bird species I’m wondering if anyone could recommend a diversity index to look at the differences in breeding bird territories between two sites. These are my data - the numbers represent the number of breeding pairs of each species:
c(4L, 3L, 14L, 1L, 4L, 11L, 2L, 1L, 2L, 7L, 1L, 5L, 7L, 3L, 1L, 2L, 1L, 1L, 4L, 1L, 1L, 4L, 2L, 4L, 3L, 3L, 1L, 1L, 1L, 14L, 3L, 14L) -> site1

c(27L, 25L, 21L, 23L, 1L, 2L, 26L, 3L, 1L, 16L, 3L, 12L, 29L, 2L, 5L, 1L, 17L, 2L, 6L, 19L, 6L, 3L, 3L, 5L, 1L, 1L, 8L, 7L, 1L, 2L, 5L, 2L, 6L, 2L, 8L, 1L, 5L, 1L, 25L, 1L, 1L, 4L, 1L, 26L, 7L, 1L, 1L, 13L, 1L, 30L, 34L, 13L, 2L) -> site2

I’ve tried Simpson’s index in R:
library(vegan)

diversity(site1, index = "simpson")
diversity(site2, index = "simpson")

But the diversity index is very similar for both sites (0.94 and 0.96 respectively) and I would say site2 is considerably more diverse than site1.
Are there any functions that are more sensitive when quantifying diversity?
 A: Firstly, I assume that you are talking about alpha-diversity (although the existence of multiple sites where the data were taken suggests that beta-diversity could be also relevant).
The simplest diversity indices (but see also here - Wikipedia has two complementary articles on the subject) are:

*

*Species richness : the number of species present, $R$.

*Shannon entropy : $H=-\sum_{i=1}^R p_i\log_2 p_i$

*Simpson index : $\lambda=\sum_{i=1}^R p_i^2$
where $p_i$ is the proportional abundance of species of type $i$ (i.e., the fraction of these species in respect to the size of the sample).
Shannon entropy and Simpson index differ from simple richness in that they give less weight to rare species, and thus less subject to fluctuations from site to site and less dependent on the sampling depth (total number of individuals sampled).
More general measures are Hill numbers
$$
^qD=\left(\sum_{i=1}^Rp_i^q\right)^{\frac{1}{q-1}},
$$
which for $q=0,1,2$ are related to the diversity indices mentioned above. Hill numbers with $q>0$ can be interpreted as effective number of species.
As a crash course about quantifying alpha-diversity, I could recommend Measuring and Estimating Species Richness, Species Diversity, and Biotic Similarity from Sampling Data by Gotelli and Chao. This also contains information on estimating the true diversity, as well as such important tools as rarefaction curves.
Finally, relative species abundance, presented by Fisher plot or Whittaker plot can be also of interest here.
Remark: As I am more a python person than an R person, I could recommend using scikit-bio, which contains library skbio.diversity, with functions for calculating alpha- and beta-diversity. Note however that most alpha-diversity metrics are easily implemented by hand.
A: The other thing that's worth taking into account (although it involves similar deep rabbit holes as the other aspects of diversity metrics mentioned in other answers and comments) is that diversity measures with low $q$ (as in @RogerVadim's answer), i.e. richness and similar metrics, are sensitive to sample size. There are 126 total breeding pairs at site 1 vs. 468 at site 2.
Here are the rarefaction curves mentioned by @RogerVadim:
library(vegan)
## site1 and site2 from OP example above
r1 <- rarecurve(rbind(site1), tidy = TRUE)
r2 <- rarecurve(rbind(site2), tidy = TRUE)
## plot site 2 first because larger sample
plot(Species~Sample, data = r2, type = "l")
with(r1, lines(Sample, Species, col = 2))


There are 32 species at site 1; if there were only 126 breeding pairs at site 2, we would expect to see only 34 or 35 species rather than the 53 actually observed.
(You can do these computations and draw the plot slightly more elegantly if your data are organized as a species-by-site matrix with zeros filled in for species that are not observed in any samples at a particular site.)
subset(r2, Sample == max(r1$Sample))

There are various ways to deal with issue; here I'm just pointing out a possible source of confounding/confusion.
A: Just about any general book on ecological methods has a section on diversity measures and there are indeed several dedicated monographs on diversity in ecology alone, to say nothing about related literatures. Ecologists can get quite agitated and even angry about which measures are good and especially about if any is best. Much of the literature is naive mathematically or scientifically, although as an outsider I won't give precise references, beyond hinting that some popular accounts are shockingly muddled.
You are too pessimistic about Simpson's measure which shows in your case for both sites a high probability of random pairs of individuals being different species.  However, small differences can be important. I suggest that the Simpson measure is usually better thought of on a different scale.
You are evidently using Simpson's measure as the complement of $R := \sum_{i=1}^S p_i^2$, namely $1 - R$ or $1 - \sum_{i=1}^S p_i^2$. Here the notation is that there are $S$ species and the typical species occurs with probability $p_i$. The $R$ notation is suggested by the term repeat rate, one of many terms in the jungle.
A reference case is $S$ species occurring equally frequently, for which  $R =
\sum_{i=1}^S p_i^2 = \sum_{i=1}^S (1/S)^2 = S/S^2 = 1/S$. This motivates $1/R$ as a numbers equivalent, namely an equivalent number of equally common species. I get that measure to be about  16.1 and   24.0 for your two sites, which is a fair contrast.
Similarly, consider Shannon entropy $H := \sum_{i=1}^S p_i \ln(1/p_i)$ which for $S$ equally common species is $S (1/S) \ln S = \ln S$. Hence $\exp(H)$ is a numbers equivalent. For your sites that gives 21.6 and  30.8.
Simplest of all is just the number of species present, $S$ itself, which for your sites are 32 and 53.
In general, $(1/R) \le \exp(H) \le S$. The measure of those three that responds best to the occurrence of rare or unusual species is number of species!
These measures are all part of a family $\sum p_i^a (\ln 1/p_i)^b$. For  Simpson's measure as a sum of squared probabilities or repeat rate $a = 2, b = 0$; for entropy $a = 1, b=1$; and for the number of species $a=0, b=0$. This easy unification was published by I.J. Good in 1953. There are fancier families associated with the names of C.R. Rao and M.O. Hill, but that Good family is simpler.
There are many other measures in the literature.
If you prefer $\log_2$ or $\log_{10}$ for calculating entropy, the corresponding numbers equivalent is then $2^H$ or $10^H$.
Note: This answer treats the data literally, as being say results from comparable sampling effort. The problem that the more you observe, the more you will find, and other more biological points, are well covered in other answers.
All that said, any idea that a single measure can capture all the information here is likely to prove disappointing. But that is true of any summary statistic.
