Checking for significant associations between a mixed set of predictors prior to running a conditional random forest model

Question

I am running a conditional random forest model using the party package in R, with the goal of quantifying variable importance (permutation importance) for 29 predictor variables. My response variable is binary (Yes/No).

The predictor variables consist of 27 categorical (mixture of binary and multi-level) and 2 continuous variables. As I am predominantly focused on variable importance, it is useful to check for multicollinearity/associations between predictors, since the inclusion of two I highly correlated/associated variables could lead to one appearing highly important, while the importance of the other is deflated (someone correct me if I'm wrong here).

The issue I have is the quantification of association between categorical and continuous predictors. Goodman and Kruskals Tau tests revealed no problematic associations between the categorical variables, while pearsons correlations revealed no issues for the continuous variables. However, Kruskal Wallis H tests revealed significant associations (alpha = 0.05) for a third of each of the pairwise comparisons between the continuous and categorical variables.

Based on the above, I have two questions:

1. When using the Kruskal-Wallis H test to check for associations between predictors, with the aim of removing highly correlated/associated variables, is 0.05 the typical removal threshold?

2. Are there other statistical approaches for testing multicollinearity/associations between a mixed set of categorical and continuous variables (with >2 levels), especially prior to running a random forest?

Many thanks.

Answer 1

Reducing the number of variables based on their correlation can make sense for improving predictive performance, but usually we are more concerned with extreme multicollinearity. Note that with any tree based model, you should also be concerned about the case where that is not the case, but where there is a one-to-one transformation that creates extreme multicollinearity (so looking at some kind of rank-based metric of association makes sense). The reason is that there's little point in including what is more or less the same information repeatedly. It just makes the trees in the RF more highly correlated, because even if you exclude one variable from constructing a tree a highly similar one might still be used, which is not a good thing (but not always a huge problem either, unless you are worried about interpreting variable importance). Common solution include eliminating variables, but also dimensionality reduction techniques (e.g. PCA, embeddings, denoising autoencoders etc.).

Additionally, as a you point out having multiple nearly identical variables can affect interpretation of variable importance (or regression coefficients in regression models). Basically, in some unpredictable manner one variable might get all the variable importance or the variable importance might be spread in some way amongst these variables making them all look not all that important. To some extent that is an unavoidable problem, because realistically there's very few variables that are not somehow associated with other variables.

However, is using some statistical test a good idea for eliminating variables? Not really. Some variables will always have at least some weak correlation and if your sample size only becomes large enough, it will end up being associated with a "significant" p-value. That does not seem right, does it, that a variable gets removed because your dataset got larger?

It may make more sense to look at diagnostics for multicollinearity (or similar) and if the metrics are extreme to make a decision. Or, one could decide on a variable of interest, and eliminate variables that are too strongly associated with that one. However, note that this does not mean that retaining the multicollinear variables would not result in a better prediction model.

Finally, note that interpreting variable importance is incredibly difficult and it is very hard to infer much from permutation importance in a way that let's you decide to do anything much. It's sort of nice to know that some variables are influencing predictions and useful in a sense, but due to various issues like associations between variables and all the known ways variable importance can be misleading one has to be rather careful with not overinterpreting it. What conclusions do you really want to draw on that basis / what recommendations do you want to give? I don't want to be too negative about these interpretation tools, but having that as a main objective for an investigation seems unusual to me.

Answer 2

I wouldn't recommend screening variables beforehand but rather use different variable importance measures (classical vs. conditional etc.) to see how they affect the outcome.

Regarding the question of testing the association between variables of different types: The natural approach in the context of conditional inference trees and forests would be to use the corresponding conditional inference test. The function independence_test() from the coin package uses the inference framework as the party and partykit packages. See also: What is the test statistics used for a conditional inference regression tree?