When to drop correlated features?

Question

I have a dataset focused on binary classification with 60 features and 5k records.

Am trying to

1) find the risk factors using statsmodel logistic regression (I do this because it's important to find risk factors that lead to a disease outcome. You might have come across several logistic regression model where they try to find the risk factors for an outcome like disease)

2) build a predictive model (I do this because not all significant risk factors are good predictors)

So to fulfill my first objective, I have to drop all correlated features to get reliable p value and coeff estimates? Am I right? Because these estimates can vary based on correlation or milticollinearity?

Lets say I have 6 features, A, B, C, D, E,F and output variables as Y. I see that A and B and C are highly correlated. So in this case, I can retain any one of these (say A) and compute the p-values. Right? Later should I again replace A with B and find out the significance? Or just finding for one feature like A, can I extend the discussion to include Other variables like B and C as well

And for second objective, I don't have to drop correlated features and rely on tree based algorithms to get best performing top n features?

Usually having correlated features don't decrease model performance like auc,accuracy etc Am I right? I mean they increase the time taken to get the result because too many features too much time to execute. Right? If A is highly correlated with B and tree based models feature importance says that A is a top performing feature, can I say that B is also a top performing feature?

Am I doing this the right way? Or do you have any suggestions on how can I do this better?

Answer 1

To build a predictive model, you must select the most relevant features in the model else if you have large number of features then your model will not converge. So will not get the right results from the model.

As you rightly mention that if features are highly correlated then the variables coefficients will be inflated.

For predictive model my suggestion to pickup the right features for your model and for that you can utilize Boruta Package in R, information values/WOE etc.