I am not a community ecologist, but these days I am working on community ecology data.

What I couldn't understand, apart from the mathematics of these distances, is the criteria for each distance to use and in what situations it can be applied. For instance, what to use with count data? How to convert slope angle between two locations into a distance? Or the temperature or rainfall at two locations? What are the assumptions for each distance and when does it make sense?

  • $\begingroup$ The reliable way to understand distance metrics, their assumptions, meaning and applicability is to meditate on their formulae. You know, comparative anatomy has let to predict how different animals live and behave. Also, read books/articles about distance metrics. $\endgroup$ – ttnphns Dec 22 '13 at 23:24
  • 3
    $\begingroup$ Pedantic note: Bray–Curtis is not a distance but a dissimilarity. $\endgroup$ – Franck Dernoncourt Dec 22 '13 at 23:29

Unfortunately, in most situations there is not a clear-cut answer to your question. That is, for any given application, there are surely many distance metrics which will yield similar and accurate answers. Considering that there are dozens, and probably hundreds, of valid distance metrics actively being used, the notion that you can find the "right" distance is not a productive way to think about the problem of selecting an appropriate distance metric.

I would instead focus on not picking the wrong distance metric. Do you want your distance to reflect "absolute magnitude" (for example, you are interested in using the distance to identify stocks that have similar mean values), or to reflect overall shape of the response (e.g. stock prices that fluctuate similarly over time, but may have entirely different raw values)? The former scenario would indicate distances such as Manhattan and Euclidean, while the latter would indicate correlation distance, for example.

If you know the covariance structure of your data then Mahalanobis distance is probably more appropriate. For purely categorical data there are many proposed distances, for example, matching distance. For mixed categorical and continuous Gower's distance is popular, (although somewhat theoretically unsatisfying in my opinion).

Finally, in my opinion your analysis will be strengthened if you demonstrate that your results and conclusions are robust to the choice of distance metric (within the subset of appropriate distances, of course). If your analysis changes drastically with subtle changes in the distance metric used, further study should be undertaken to identify the reason for the inconsistency.

  • 1
    $\begingroup$ What do you mean by correlation distance? 1-r? $\endgroup$ – ttnphns Dec 23 '13 at 9:38
  • 1
    $\begingroup$ @ttnphns yep, $1-r$ is most common. It's worth noting that for a given similarity metric $\rho \in [-1,1]$ there are at least three formulas for converting to a dissimilarity: (1) Bhattacharyya's method $cos^{-1}(\rho)$, (2) Kolmogorov's method $1-\rho$, and (3) Matusita's method $\sqrt{2-2\rho}$. This is another area where in $practice$ I don't think the choice usually matters much, and if it did, I would be concerned about the robustness of my results. $\endgroup$ – ahfoss Dec 23 '13 at 16:32
  • $\begingroup$ Citation for my last comment: Krzanowski (1983). Biometrika, 70(1), 235--243. See page 236. $\endgroup$ – ahfoss Dec 23 '13 at 16:33
  • 1
    $\begingroup$ OK, thanks. Check also this answer please. It poins to the fact that r is exactly related to euclidean distance obtained on the standardized data (profiles being compared), which reflect overall shape of the response in your words. $\endgroup$ – ttnphns Dec 23 '13 at 17:43
  • 1
    $\begingroup$ Good post. The two metrics are indeed related, as you point out. To contextualize your points to the current discussion, the key difference is that in Euclidean distance variables are not (usually) centered, but the correlation formula centers variables and scales by their standard deviation. Thus, correlation is invariant to linear transformations, while Euclidean distance is not necessarily. $\endgroup$ – ahfoss Dec 23 '13 at 17:53

Choosing the right distance is not an elementary task. When we want to make a cluster analysis on a data set, different results could appear using different distances, so it's very important to be careful in which distance to choose because we can make a false good artefact that capture well the variability, but actually without sense in our problem.

The Euclidean distance is appropriate when I have continuous numerical variables and I want to reflect absolute distances. This distance takes into account every variable and doesn’t remove redundancies, so if I had three variables that explain the same (are correlated), I would weight this effect by three. Moreover, this distance is not scale invariant, so generally I have to scale previously to use the distance.
Example ecology: We have different observations from many localities, of which the experts have taken samples of some microbiological, physical and chemical factors. We want to find patterns in ecosystems. These factors have a high correlation, but we know everyone is relevant, so we don’t want to remove these redundancies. We use the Euclidean distance with scaled data to avoid the effect of units.

The Mahalanobis distance is appropriate when I have continuous numerical variables and I want to reflect absolute distances, but we want to remove redundancies. If we have repeated variables, their repetitious effect will disappear.

The family Hellinger, Species Profile and Chord distance are appropriate when we want to make emphasis on differences between variables, when we want to differentiate profiles. These distances weights by total quantities of each observation, in such a way that the distances are small when variable by variable the individuals are more similar, although in absolute magnitudes was very different. Watch out! These distances reflect very well the difference between profiles, but lost the magnitude effect. They could be very useful when we have different sample sizes.
Example ecology: We want to study the fauna of many lands and we have a data matrix of an inventory of the gastropod (sampling locations in rows and species names in columns). The matrix is characterized by having many zeros and different magnitudes because some localities have some species and others have other species. We could use Hellinger distance.

Bray-Curtis is quite similar, but it’s more appropriate when we want to differentiate profiles and also take relative magnitudes into account.

  • $\begingroup$ Thank you for differenciating the use cases and examples. Found this very helpful in application to an aero classification model. $\endgroup$ – S3DEV Apr 14 '20 at 9:02

Regarding the Manhattan distance: Kaufman, Leonard, and Peter J. Rousseeuw. "Finding groups in data: An introduction to cluster analysis." (2005).

The use of the Manhattan distance is advised in those situations where for example a difference of 1 in the first variable,and of 3 in the second variable is the same as a difference of 2 in the first variable and of 2 in the second.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.