When to drop correlated features?

Question

I have a dataset focused on binary classification with 60 features and 5k records.

Am trying to

1) find the risk factors using statsmodel logistic regression (I do this because it's important to find risk factors that lead to a disease outcome. You might have come across several logistic regression model where they try to find the risk factors for an outcome like disease)

2) build a predictive model (I do this because not all significant risk factors are good predictors)

So to fulfill my first objective, I have to drop all correlated features to get reliable p value and coeff estimates? Am I right? Because these estimates can vary based on correlation or milticollinearity?

Lets say I have 6 features, A, B, C, D, E,F and output variables as Y. I see that A and B and C are highly correlated. So in this case, I can retain any one of these (say A) and compute the p-values. Right? Later should I again replace A with B and find out the significance? Or just finding for one feature like A, can I extend the discussion to include Other variables like B and C as well

And for second objective, I don't have to drop correlated features and rely on tree based algorithms to get best performing top n features?

Usually having correlated features don't decrease model performance like auc,accuracy etc Am I right? I mean they increase the time taken to get the result because too many features too much time to execute. Right? If A is highly correlated with B and tree based models feature importance says that A is a top performing feature, can I say that B is also a top performing feature?

Am I doing this the right way? Or do you have any suggestions on how can I do this better?

Answer 1

"Significant correlation" would usually mean that you tested a null hypothesis that $\rho=0$. Depending on your sample size, such correlation may still be quite close to zero. Why would you drop variables that have a very small correlation? Moreover, correlation measures a linear relationship between variables. Below you can see examples of correlation coefficients for different variables, including ones with non-linear relations.

Say that you have three features $A$ is correlated with $B$, $B$ is correlated with $C$, but not with $A$, where by "is correlated" I mean here that they are higher than some arbitrary threshold. Which ones would you drop? If you drop $C$ first, you'll need to drop also $B$, but if you drop $B$ first, you will leave $C$. Or maybe you drop $A$ and $B$, or $A$ and $C$? Say that you want to predict if someone is going to be arrested, you have two features: "has a tattoo" and "spend time in prison" that are correlated, would it be reasonable to drop "spend time in prison" because it is correlated with "has a tattoo"? Obviously, the causal relation is the other way around and having a tattoo is only a proxy for measuring the latter feature.

Another problem is that with looking at the correlation matrix you consider only pairs of features. It can be the case that two (or more) features are not enough if you consider each of them separately, but they interact, so taken together they would be meaningful. In simple models like linear regression, you would need to consider interactions explicitly, but many machine learning models would learn the interaction by themselves. Looking at individual correlations you may accidentally drop such features.

If you have many features, you can use regularization instead of throwing away data. In some cases, it will be wise to drop some features, but using something like pairwise correlations is an overly simplistic solution that may be harmful.

Answer 2

To build a predictive model, you must select the most relevant features in the model else if you have large number of features then your model will not converge. So will not get the right results from the model.

As you rightly mention that if features are highly correlated then the variables coefficients will be inflated.

For predictive model my suggestion to pickup the right features for your model and for that you can utilize Boruta Package in R, information values/WOE etc.