why model perform slighty worst after removing co-related features I have a classification problem on which I am testing the main classification models like Logistic Regression, SVM, KNN and deep neural networks.
I have a feature set of 40.
And around 5-6 are highly co-related with value >=.9 or <=-.9
To my surprise, when I am removing these co-related variables, the performance slightly gets bad on test data.
Now, as per my theoretical knowledge, removing correlated features should remove noise and this improves performance.
Upon, googling I found 1 article which pointed out to drop only those which are not co-related with output result. I tried that too but still not luck. Performance reduces slightly after dropping those features.
As I am new to data science, can someone guide me on where I could be understanding or going wrong.
P.S : I am not sharing data information or implementation details, as I am interested in first knowing the possibility of this case.
 A: Putting aside the excellent points made elsewhere regarding how you define accuracy, whether it’s truly out of sample accuracy you’re testing etc etc, there’s also the point that multicollinearity isn’t necessarily a bad thing for prediction accuracy. It can be a nightmare for understanding predictions, eg in simple multivariate regression it can make it impossible to interpret coefficients. But being a problem for interpretation is not the same as being a problem for prediction.
So what you may well be seeing is that, while they are correlated, they might still be providing some independent information that your model can use for helping it train. In other words they’re still useful for prediction accuracy. That means it’s not surprising that you see a reduction in prediction accuracy by removing these. 
A: Setting aside what Cross Validated tends to think of classifier metrics that depend on thresholds,$^{\dagger}$ I think I see your problem.
We tends to care very little about machine learning performance on in-sample data. Adding more and more parameters allows us to play connect the dots (so to speak) and memorize the training data. 
What we care about is how the machine learning model performs on unseen data, since this mimics how real machine learning works (e.g. Apple or Amazon doing speech recognition on sentences that have yet to be spoken). Apply your models to data that you've held out from the training data, and see if you get the same issue of the simpler model having higher F1 score.
$^{\dagger}$See, for instance, my post from the past few weeks that has an excellent answer to an issue that a practicing data scientist may encounter. (TLDR: look at the predicted probabilities, not the classifications based on a particular threshold.) There are lots of other posts on CV about these "proper scoring rules", too.
