Discordance between various methods of multivariate outliers detection

Question

Here is a small "toy example" dataset, with 15 individuals described by 6 variables (this is R language):

dat <- structure(list(V1 = c(82L, 72L, 80L, 72L, 79L, 83L, 76L, 80L, 
85L, 74L, 89L, 80L, 90L, 63L, 70L), V2 = c(88L, 93L, 104L, 89L, 
83L, 103L, 87L, 101L, 95L, 88L, 99L, 94L, 97L, 87L, 87L), V3 = c(98L, 
95L, 97L, 95L, 86L, 97L, 89L, 97L, 91L, 85L, 99L, 90L, 99L, 75L, 
87L), V4 = c(91L, 97L, 87L, 80L, 80L, 96L, 80L, 101L, 86L, 77L, 
100L, 93L, 98L, 74L, 87L), V5 = c(76L, 65L, 65L, 67L, 75L, 69L, 
71L, 71L, 74L, 66L, 68L, 67L, 78L, 58L, 66L), V6 = c(109L, 99L, 
105L, 104L, 87L, 103L, 99L, 98L, 111L, 97L, 97L, 92L, 104L, 82L, 
87L)), class = "data.frame", row.names = c(NA, 15L))
# Show data:
dat

Lets say that the variables are some osteometric measurements (bone lengths or widths, or whatever), and therefore express the global sizes of the individuals. In this small sample, there is no huge outlier, but some slightly unusual values. A PCA can be performed so that you can see the general structure of the data:

FactoMineR::PCA(dat)

I applied two methods of multivariate outliers detection: robust Mahalnobis distance, and isolation forests. The individuals detected as potential outliers are totally different from one method to the other, and I would like to know why (or which is the global "philosophy" of each method).

1. Isolation forests

Here, I use the R package solitude that implements a fast algorithm for isolation forests:

library(solitude)
isofo <- isolationForest$new(sample_size = nrow(dat), seed = 2020)
isofo$fit(dat)
# Anomaly scores by increasing order:
scores <- isofo$scores$anomaly_score
names(scores) <- rownames(dat)
sort(scores)

As you can see by running this R code, the individual #14, having the highest anomaly score, is the best candidate for being an outlier. It corresponds to the "smallest" individual in this dataset, with relatively low values for all variables (and was the leftmost individual on PC1 axis, which was related to size).

2. Robust Mahalanobis distance (a.k.a. MCD estimator)

Here, I use the R package robustbase, and I compute the robust Mahalnobis distance for each individual, with a parameter $\alpha = 0.75$ as suggested in the original publication:

library(robustbase)
mcd <- covMcd(dat, alpha = 0.8)$mah
names(mcd) <- rownames(dat)
sort(mcd)

According to those distances, the individuals #9 and #11 are from far the most credible candidates for being outliers.

3. A possible exaplanation?

Here, isolation forests seem to be more sensitive to anomalies in sizes, and detect an individual one could describe as "small" compared to all other individuals.

On the other hand, robust Mahalnobis distances primarily detect individuals showing anomalies of "shape", i.e. whose values do not respect the correlation pattern usually observed among the variables: individual #9 has a high value of V6 given the values of the other variables (i.e., its global "size"), and individual #11 has a high value of V2 and V4 given its global size.

Is this correct? It seems that one should be really careful in choosing one method or the other, depending on his/her own definition of what is an "anomaly"/outlier in the dataset. Is the discordance observed on this small sample a simple and meaningless artifact, or a real difference of "philosophy" between the two methods?

Answer 1

Mahalanobis distance only considers linear relationships between variables. As long as you are considering multivariate data that has simple relationships, with no polinomial dependency or non-linearity, it works well, but otherwise it does not.
Therefore, I would strongly discourage its use in most Machine Learning applications!

Example:
I sampled this data in a way that it has no linear correlation, and the two variables are standardized. Therefore, the Mahalanobis distance for this sample is equivalent to the Euclidean distance. If I insert an outlier close to the origin (clearly an outlier to the human eye, and not just for its color :) ) this will have the SMALLEST mahalanobis distance of all the points, as it is the closest to the centroid (black), while points in the corners will have high scores even though they clearly follow the same pattern as the rest of the data.
Indeed, the centroid means very little for the distribution when the data does not have a simple linear dependency!

Isolation forest on the other hand makes no assumptions on the distribution, and gives a high score to points that are easier to isolate by randomly splitting on the variables.

In general, Isolation Forest is much more reliable, even though I would not necessarily use it for particularly small datasets like the one you have shown, where other methods such as hierarchical clustering might work better. Mahalanobis Distance (also in its Robust version) should only be used if the data has very simple structure and distrbution (also, I think it needs to do the inversion of the correlation matrix, so I would avoid using it for big datasets!)