I need an appropriate method for quantifying spatial spread (i.e., clustering), where area of interest is the country of India divided into 25km$\times$25km grid cells, and the response variable is a count at the 25km$\times$25km resolution (number of something, in my case, eBird (birdwatching) checklists; or some proportion of the same). After some extensive searching, I arrived at a method that looked promising, but ran into a small but significant issue. Hence, I would like advice on how to tackle the issue, or suggestions on alternative methods.
The analytical method I found is Cluster and Outlier Analysis, and the most common approach for this is using Moran's I.
- Looking into Cluster and Outlier Analysis led me to Moran's I, which is a correlation coefficient that measures the overall spatial autocorrelation of your data set. It ranges from -1 to +1 (albeit not perfectly), but unlike other correlation coefficients that measure perfect correlation to no correlation, Moran's I measures perfect dispersion (which is clustering of dissimilar values, -1), perfect clustering (which is clustering of similar values, +1) and perfect randomness (no autocorrelation, 0).
- Moreover, unlike other correlation coefficients, you can't take Moran's I index at face value. It is an inferential statistic, so instead you need to determine statistical significance in order to interpret the results. For this, a simple hypothesis test is sufficient, giving a z-score and its associated p-value. The null hypothesis is that the data is randomly distributed, while the alternative is that the data is more clustered than expected by chance. Within this latter hypothesis, two scenarios are possible based on the z-score: positive implies spatial clustering while negative implies dispersion (where likes could be repelling likes).
- Such statistics of global spatial autocorrelation tell us only about the complete (i.e., global) spatial pattern. Thus, the concept of local indicator of spatial association (LISA) was introduced by anselin1995. A LISA has two important features: first, it provides a statistic for each location (a local spatial unit) with a significance assessment; second, it establishes a proportional relationship between the sum of the local statistics and a/the corresponding global statistic.
- One such LISA, Local Moran, was suggested by anselin1995 as a way to identify local clusters and outliers. The sum of the local statistics is proportional to the global Moran’s I, or, alternatively, the global Moran’s I corresponds with the average of the local statistics. To assess significance, a conditional permutation method is advised. However, significance in and of itself is not very useful for Local Moran. When the significance information is combined with location information, it allows a very powerful interpretation---classifying the significant locations as HH & LL spatial clusters and HL & LH spatial outliers (i.e., cluster and outlier analysis). This is a bit more advanced than simple hotspot analysis.
My aim is to obtain, for India, both a global spatial clustering measure (global Moran) as well as local clusters and outliers (local Moran), for a certain time period. Then, these measures can be compared statistically (in the case of global Moran) and visually (local Moran) between different time periods.
Theoretically, this analysis sounds very promising and the output metric is rather easy to interpret. However, the problem is that Moran's I uses z-scores, which in turn requires the input data to be normally distributed. Therefore, although I was able to run the analyses and produce some outputs, the nature of the count data with high right skew (even after removing 0s) and a very long tail means that the identification of clusters and outliers is not very meaningful (see attached figure where only HH and LH are identified).
I tried several types of transformations on the count data: $log(n)$, $sqrt(n)$, $log(n+1)$, $log(n+c)$ (from discussion here), Box-Cox (see here; but this doesn't work with zeroes in original data), Yeo-Johnson (first comment here), and inverse hyperbolic sine (see here)---but all to no avail. The data just does not seem to be cooperating, and I don't know how else to normalise the data. (See attached histogram of the data without any transformation.)
griffith2010 concluded that high sample size is a solution for non-normal data in Moran’s calculations. This is clearly not the case for me: eBird is a citizen science project that gathers data on birds from birdwatchers everywhere; as such, the sample sizes I am working with are much higher than expected in conventional studies.
From other reading on CrossValidated and elsewhere, I did come across chen2013 and dutilleul2000 discussing Moran's I in relation to Geary's C and Mantel's test, but they were mathematically heavy and I didn't take away any major relevant points for my particular issue. Similarly, I found a discussion on saddlepoint approximation in the context of Moran's I, and although this may have some useful information for me I was unable to grasp it well. (Hence, since this is the only lead that seemed somewhat promising, I would appreciate clarification of this as well!)
I tried searching for Generalised forms of Moran's I or some similar analysis of clusters and outliers, but could not find any leads. A popular alternative to Moran's I is Geary's C, but this is just different in sensitivity to local clusters vs outliers, and is still reliant on z-scores and normality.
So, any ideas on how I can possibly tackle this issue? I assume cluster analysis of (non-normal) count data is not uncommon, so there must surely be some approach that overcomes the problem of normality? Are there any better, generalised methods of Cluster and Outlier Analysis? Or, if nothing, some completely different approach to quantifying spatial spread/clustering?
I've also left some more links on Moran's I below.
Other links on Moran's I
Cluster and Outlier Analysis using Moran's I failing: only HH and LH being detected, while all others classified as non-significant. While I do expect some NS cells, there should certainly be all four classes of clusters and outliers as well.
Histogram of untransformed data showing non-normality. I'm also curious about histograms of data in cases where Moran's I has worked well---although they are count variables, they don't seem to be so problematic!