Linear correlation and machine learning methods

Question

I am reading a paper that has a wide number of explanatory variables. In order to reduce this number, the author first computes the (Pearson) correlation between each pair and keeps just one of the high correlated ones. This way, he reduced the number of variables and then feed an ML model with it.

My question: from what I understand, there are some ML techniques that 'are able' to identify no linear relationships. Is it ok to employ linear correlation to dismiss some variables because of their LINEAR relationship to then use a model which not necessarily models a linear relationship?

Can somebody help me to clarify this doubt? Thank you!

Answer 1

Consider the example below.

set.seed(2023)
N <- 1000
x1 <- runif(N, -2, 2)
x2 <- runif(N, -2, 2)
x3 <- runif(N, -2, 2)
y <- 0*x1 + 0*x2 + 0*x3 + x3^2
cor(x1, y) # -0.02706921
cor(x2, y) # 0.02323476
cor(x3, y) # -0.001507549

Here, there are three possible variables (x1, x2, x3) to predict the outcome (y). The correlation between each predictor variable and y is close to zero and would be eliminated by your proposed method of screening out variables that have a low correlation with y. However, squaring x3 is a perfect predictor of y. If you have a flexible model that can catch these kinds of nonlinear functions of the original data, you are depriving the model of data that could be highly predictive when used the right way (which a good model should figure out).

For instance, a neural network is able to detect the nonlinear relationship between x3 and y without being explicitly programmed to look for such a relationship, yet screening based on the correlation would deprive such a model of that crucial x3 variable.

library(nnet)
L <- nnet::nnet(y ~ x1 + x2 + x3, size = 3, linout = T)
plot(x3, y)
lines(x3, predict(L))

(Yes, there are alternatives to the Pearson correlation used here. Spearman correlation could be an alternative, though the Spearman correlations here are all quite low, too, and would not be particularly helpful.)

Is it ok to employ linear correlation to dismiss some variables because of their LINEAR relationship to then use a model which not necessarily models a linear relationship?

The example above hopefully demonstrates how such an approach can do serious harm to your analysis.