2
$\begingroup$

I have a classification problem on which I am testing the main classification models like Logistic Regression, SVM, KNN and deep neural networks.

I have a feature set of 40.

And around 5-6 are highly co-related with value >=.9 or <=-.9

To my surprise, when I am removing these co-related variables, the performance slightly gets bad on test data.

Now, as per my theoretical knowledge, removing correlated features should remove noise and this improves performance.

Upon, googling I found 1 article which pointed out to drop only those which are not co-related with output result. I tried that too but still not luck. Performance reduces slightly after dropping those features.

As I am new to data science, can someone guide me on where I could be understanding or going wrong.

P.S : I am not sharing data information or implementation details, as I am interested in first knowing the possibility of this case.

Improve this question
$\endgroup$
4
  • $\begingroup$ Worse on what and in what way? For instance, are you measuring accuracy on training data or ROC-AUC or Brier score on out-of-sample data? $\endgroup$
    – Dave
    Commented Jun 4, 2020 at 16:11
  • $\begingroup$ based on f1 score of test data for both cases where no correlation is done and where correlated features are removed $\endgroup$
    – Onki
    Commented Jun 4, 2020 at 16:12
  • $\begingroup$ On in-sample or out-of-sample data? And F1 score depends on your threshold for classification. Perhaps the models with fewer variables have better F1 scores at most thresholds, just not the default of $0.5$ (or whatever you used). $\endgroup$
    – Dave
    Commented Jun 4, 2020 at 16:14
  • $\begingroup$ in-sample test data. threshold remains .5. But is it among the standard practices to change the threshold if you are removing correlated features. I am sorry if my qsn looks silly. I am trying to learn this subject newly. $\endgroup$
    – Onki
    Commented Jun 4, 2020 at 16:16

2 Answers 2

Reset to default
3
$\begingroup$

Putting aside the excellent points made elsewhere regarding how you define accuracy, whether it’s truly out of sample accuracy you’re testing etc etc, there’s also the point that multicollinearity isn’t necessarily a bad thing for prediction accuracy. It can be a nightmare for understanding predictions, eg in simple multivariate regression it can make it impossible to interpret coefficients. But being a problem for interpretation is not the same as being a problem for prediction.

So what you may well be seeing is that, while they are correlated, they might still be providing some independent information that your model can use for helping it train. In other words they’re still useful for prediction accuracy. That means it’s not surprising that you see a reduction in prediction accuracy by removing these.

Improve this answer
$\endgroup$
0
$\begingroup$

Setting aside what Cross Validated tends to think of classifier metrics that depend on thresholds,$^{\dagger}$ I think I see your problem.

We tends to care very little about machine learning performance on in-sample data. Adding more and more parameters allows us to play connect the dots (so to speak) and memorize the training data.

What we care about is how the machine learning model performs on unseen data, since this mimics how real machine learning works (e.g. Apple or Amazon doing speech recognition on sentences that have yet to be spoken). Apply your models to data that you've held out from the training data, and see if you get the same issue of the simpler model having higher F1 score.

$^{\dagger}$See, for instance, my post from the past few weeks that has an excellent answer to an issue that a practicing data scientist may encounter. (TLDR: look at the predicted probabilities, not the classifications based on a particular threshold.) There are lots of other posts on CV about these "proper scoring rules", too.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ TLDR: look at the predicted probabilities, not the classifications based on a particular threshold.) -> I will look into this and then give an update here. $\endgroup$
    – Onki
    Commented Jun 4, 2020 at 16:33
  • 1
    $\begingroup$ I will admit that using proper scoring rules is a more advanced topic. I suggest focusing on testing your models on out-of-sample data before you confuse yourself a bunch with proper scoring rules, even if they’ve preferable. $\endgroup$
    – Dave
    Commented Jun 4, 2020 at 16:39
  • $\begingroup$ I assume by out of sample data you mean on test data? bcoz getting more data apart the dataset given is out of question. This only I can divide in train and test data. N I am already testing this on test data only $\endgroup$
    – Onki
    Commented Jun 4, 2020 at 17:12
  • $\begingroup$ "this mimicks how real machine learning works" lol $\endgroup$
    – carlo
    Commented Jun 4, 2020 at 18:04
  • $\begingroup$ @Onki You said earlier that you're working on in-sample data. Are you doing some kind of train_test_split in Python's sklearn for instance? That gives you out-of-sample data for testing your model. If you're doing something like that and still getting funky F1 behavior, then the answer by mooks applies. Sure, two variables may be correlated, but unless the correlation is perfect (e.g. reporting distance in both feet and meters), each variable has some information that the other does not... $\endgroup$
    – Dave
    Commented Jun 4, 2020 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.