I have a dataset focused on binary classification with 60 features and 5k records.
Am trying to
1) find the risk factors using statsmodel logistic regression (I do this because it's important to find risk factors that lead to a disease outcome. You might have come across several logistic regression model where they try to find the risk factors for an outcome like disease)
2) build a predictive model (I do this because not all significant risk factors are good predictors)
So to fulfill my first objective, I have to drop all correlated features to get reliable p value and coeff estimates? Am I right? Because these estimates can vary based on correlation or milticollinearity?
Lets say I have 6 features, A, B, C, D, E,F and output variables as Y. I see that A and B and C are highly correlated. So in this case, I can retain any one of these (say A) and compute the p-values. Right? Later should I again replace A with B and find out the significance? Or just finding for one feature like A, can I extend the discussion to include Other variables like B and C as well
And for second objective, I don't have to drop correlated features and rely on tree based algorithms to get best performing top n features?
Usually having correlated features don't decrease model performance like auc,accuracy etc Am I right? I mean they increase the time taken to get the result because too many features too much time to execute. Right? If A is highly correlated with B and tree based models feature importance says that A is a top performing feature, can I say that B is also a top performing feature?
Am I doing this the right way? Or do you have any suggestions on how can I do this better?