Statistical test to perform on species diversity / Simpson's diversity index

Question

I have a set of data from each of the two sites I am measuring for plant species diversity. I plan on using the Simpson's diversity index (SDI), which combines species richness (number of different species) with the number of each individual to form a number between 0 and 1. The mean for the SDI value for each site will be calculated from the different samples at each site.

What statistical test should I use for comparing the values for the two sites? I've heard that a t-test may be useful but I'm not sure.

Answer 1

Before beginning any sort of statistical analysis, I would strongly consider transforming your data from diversity indices into effective species counts. Indexes such as Simpson's and Shannon's are highly nonlinear (e.g., a doubling of the index value does not equal a doubling of species diversity). This could present a substantial challenge to interpreting your results. For example, a Simpson's index with a standard deviation of 0.1 represents substantially more variation in actual diversity when it the mean is near 0 than when it is near 1. Additionally, the arithmetic mean of multiple diversity indices will not provide ecologically meaningful results.

Using effective species counts (which are explained simply and thoroughly in the above link) linearizes these measurements, which will make the subsequent analyses substantially easier to interpret.

Answer 2

Using the sbdiv function from the simboot R package will allow you to test for differences between sites without having to pool data from your different samples within one site.

Two difficulties in this situation make the t-test without any correction highly unsuitable. First, unless you have a large number of observations (sites/quadrats sampled) per habitats, you might need to use non-parametric tests, as the assumptions for using Student's t-test (normal distribution) or the Normal approximation for estimating the mean of large samples are not met.

Then, indices include a level of uncertainty that can vary from one sample to the next depending on the number of individuals and species/taxa found in each. Try estimating the confidence interval of your indices for each sample through bootstrapping just to understand what I mean. When performing a t-test, strong assumptions are made on this uncertainty which are probably not met with these indices, and in particular that it is the same for each sample. Check the litterature on meta-analyses (where summary statistics are routinely used in analyses) to get a better understanding of the problem.

Answer 3

I think there are two really critical questions to answer here. First is what are you really trying to measure? The second is once you've determined that, what are principled and feasible ways to compare your measurements from the two communities? I will try to address both a bit below.

Simpson’s Index as defined by Simpson in 1949 is a probability that as you state, ranges between 0 and 1. Specifically, it is an estimate of the probability that in the entire community you sampled, two randomly drawn individuals would have the same identity. Of the many important things to note, one is that therefore higher values imply a lower diversity. A second is that there are a few ways to compute Simpson based on your data. Sum over all species(relative abundance of species ^2) is that probability of two randomly drawn individuals being the same species in the population, but likely you are interested in estimating that from a subsample of that population. Simpson provided a finite-community-size unbiased estimator (Simpson 1949), and an accompanying variance estimator. To the best of my knowledge, these are valid. So one principled answer to your question is to use Simpson’s estimator of Simpson’s index with Simpson’s variance estimator, and then to determine the chance that those two values are “drawn from the same distribution” based on e.g. the overlap in the CIs from those variance estimators with your favorite statistical test.

As C.R. Peterson alluded to, there are many reasons why you might not actually want to use Simpson’s index if you are interested in comparing diversity. Lou Jost’s website, which Peterson linked to, has an excellent description of some of the issues with the raw Simpson index and an argument for using Hill numbers, which instead compute the average species rarity of each community… the Simpson analog is the harmonic mean rarity, where each species rarity (1/relative abundance) is weighted by its relative abundance. In this case, you might use Chao and Jost’s bootstrapped confidence intervals https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12349, which you can implement yourself, or use iNEXT (available as web tool online or as an R package) http://chao.stat.nthu.edu.tw/wordpress/software_download/inext-online/.

One thing that is blowing my mind and brought me to your post is that whereas Simpson’s finite size estimate is both unbiased and well constrained/accurate at very low sample sizes, the sampling variance around the Hill number version (the reciprocal of Simpson’s index, sometimes called “Simpson’s Diversity” by Jost and colleagues) can remain really large even for giant sample sizes. Additionally, to the best of my knowledge, although Chao and Jost’s estimator is quite good, there is no known unbiased estimator on the diversity/effective numbers/ Hill scale for Simpson’s diversity. So there is definitely some tradeoff in what you can say about your estimates in terms of interpretation vs. what you can say in terms of estimation. I would love to be more precise in this regard but still have some learning to do.

Answer 4

From the description, a t-test is the way to go. To rephrase what you said, you have two sites, and multiple observations at each site. This is the classic example of a two-sample independent T-Test (wiki).

The gist of it is you need to not only see how far apart the means are, you need to know how variable the data is. That way you can say the difference in means is unlikely to be attributable to random sampling variance. The t-test tells you exactly that.

You can do a t test in excel as fast as any of the stats programs. You just need two columns of SDI values, one for each site. Then you use the function t.test(), select your first column for the first parameter, the second column for your second parameter, use a 2-tailed test (wiki) for the third parameter (enter the number 2), and use the second option for the last parameter (homoskedastic).

If you use R, the function is also t.test(), but there are multiple ways you can set up your data. Here is a tutorial. Basically, you could have two columns, one for each site like Excel, or one column with all your SDI values, and another column that categorizes your data (i.e. tells you whether that row was a sample from Site A or Site B).

There are a lot of other considerations, for instance your samples from your sites ideally would have similar standard deviations. If they are very different, you can use option 3 for the last parameter in Excel. I won't give a full step-by-step in R, but there are plenty of ways to test for differences in standard deviation and correct for it. However, basic t-tests assuming homogeneity of variance (i.e. what I described above) should get you going.