Datasaurus Dozen and (correlated) feature importance?

Question

For a bit of context of where I am coming from, I suggest reading this other question I have asked on CV.

When dealing with feature importance, especially when using random forests or other tree-based models, a point of interest is always the bias of the importance metric towards correlated features, with plenty of literature around the topic and various proposed algorithms that deal with correlation/collinearity by removing these biases.

However, as a tought experiment, what would happen when such measures are in place and we encounter a hypothetical dataset made up of the same data as the thirteen data sets in the Datasaurus Dozen (which, as far as I am aware, are thirteen completely different sets of data)?

Such a dataset would have 26 features (13 $X$ columns and 13 $Y$ columns from the datasaurus), however the pairwise correlation among all of the $X$ columns and among all of the $Y$ columns would be very close to 1, since the mean values of those columns is the same up to the second decimal place.

In such an hypothetical scenario, even though the data comes from thirteen different data sets, techniques such as drop column importance or conditional permutation importance would output an importance score which is very close, if not exactly, 0 for all of the features.

From the "importance with respect to the model" perspective, this would even be correct, if all of the columns provide the same information to the model, no column is inherently important to it. However to us humans interested in knowing which feature/variable is of importance in a classification/prediction task, this would be worthless.

Am I correct in my though experiment? Is this what would happen and is this what is expected out of these algorithms? If it's not, what would happen and would these algorithms be resilient to such situations? To me it seems like methods which do not concern themselves with removing a bias towards correlated features would actually fare better here, and give an output which would be somewhat useful to a human.

Answer 1

The median pairwise correlation among the $X$ variables is 0.86, and among the $Y$ variables is 0.94. But we could shuffle the rows separately for each of the 12 component datasets and reduce the correlation to nearly zero without affecting the fact that the means, variances, and correlations are identical in each dataset.

So, this example doesn't really help with discussions about collinearity.

Update: I will also note that there's a reason the correlation is quite high, but it doesn't help with arguments about collinearity. The correlation is high so that animations smoothly transitioning between the twelve datasets don't have to move the points too far.