Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am not a community ecologist, but these days I am working on community ecology data.
What I couldn't understand, apart from the mathematics of these distances, is the criteria for each distance to use and in what situations it can be applied. For instance, what to use with count data? How to convert slope angle between two locations into a distance? Or the temperature or rainfall at two locations? What are the assumptions for each distance and when does it make sense?
Unfortunately, in most situations there is not a clear-cut answer to your question. That is, for any given application, there are surely many distance metrics which will yield similar and accurate answers. Considering that there are dozens, and probably hundreds, of valid distance metrics actively being used, the notion that you can find the "right" distance is not a productive way to think about the problem of selecting an appropriate distance metric.
I would instead focus on not picking the wrong distance metric. Do you want your distance to reflect "absolute magnitude" (for example, you are interested in using the distance to identify stocks that have similar mean values), or to reflect overall shape of the response (e.g. stock prices that fluctuate similarly over time, but may have entirely different raw values)? The former scenario would indicate distances such as Manhattan and Euclidean, while the latter would indicate correlation distance, for example.
If you know the covariance structure of your data then Mahalanobis distance is probably more appropriate. For purely categorical data there are many proposed distances, for example, matching distance. For mixed categorical and continuous Gower's distance is popular, (although somewhat theoretically unsatisfying in my opinion).
Finally, in my opinion your analysis will be strengthened if you demonstrate that your results and conclusions are robust to the choice of distance metric (within the subset of appropriate distances, of course). If your analysis changes drastically with subtle changes in the distance metric used, further study should be undertaken to identify the reason for the inconsistency.
correlation distance? 1-r?
$\endgroup$
– ttnphns
Dec 23 '13 at 9:38
reflect overall shape of the response in your words.
$\endgroup$
– ttnphns
Dec 23 '13 at 17:43
Choosing the right distance is not an elementary task. When we want to make a cluster analysis on a data set, different results could appear using different distances, so it's very important to be careful in which distance to choose because we can make a false good artefact that capture well the variability, but actually without sense in our problem.
The Euclidean distance is appropriate when I have continuous numerical variables and I want to reflect absolute distances. This distance takes into account every variable and doesn’t remove redundancies, so if I had three variables that explain the same (are correlated), I would weight this effect by three. Moreover, this distance is not scale invariant, so generally I have to scale previously to use the distance.
Example ecology: We have different observations from many localities, of which the experts have taken samples of some microbiological, physical and chemical factors. We want to find patterns in ecosystems. These factors have a high correlation, but we know everyone is relevant, so we don’t want to remove these redundancies. We use the Euclidean distance with scaled data to avoid the effect of units.
The Mahalanobis distance is appropriate when I have continuous numerical variables and I want to reflect absolute distances, but we want to remove redundancies. If we have repeated variables, their repetitious effect will disappear.
The family Hellinger, Species Profile and Chord distance are appropriate when we want to make emphasis on differences between variables, when we want to differentiate profiles. These distances weights by total quantities of each observation, in such a way that the distances are small when variable by variable the individuals are more similar, although in absolute magnitudes was very different. Watch out! These distances reflect very well the difference between profiles, but lost the magnitude effect. They could be very useful when we have different sample sizes.
Example ecology: We want to study the fauna of many lands and we have a data matrix of an inventory of the gastropod (sampling locations in rows and species names in columns). The matrix is characterized by having many zeros and different magnitudes because some localities have some species and others have other species. We could use Hellinger distance.
Bray-Curtis is quite similar, but it’s more appropriate when we want to differentiate profiles and also take relative magnitudes into account.
Regarding the Manhattan distance: Kaufman, Leonard, and Peter J. Rousseeuw. "Finding groups in data: An introduction to cluster analysis." (2005).
The use of the Manhattan distance is advised in those situations where for example a difference of 1 in the first variable,and of 3 in the second variable is the same as a difference of 2 in the first variable and of 2 in the second.
