I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.

When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th & 8th) feature, I see improvement only in one of the models (LR scikit shows improvement in AUC for 7th feature only whereas Xgboost shows improvement when 7th and 8th features are added). I also understand that AUC may not be the reliable metric, So I can use log loss for logistic regression. But how do I use one metric which can help me compare the performance of all models? Similar to log loss, is there any other metric for other algorithms like Random Forest, Xgboost, SVM etc?

I also see that we have cochran's Q test which can help us know whether classifiers are different or not? But can this help in picking one classifier over another?

enter image description here

Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models? How should I interpret this?

Please note that I split the data into train and test and did 10 fold CV on train data.

So, how do I know that this newly added features are really helping in improving the model performance? Is there any statistic to find this?

Can help me with this?

  • $\begingroup$ I would use an out of sample data set and see which model (regardless of which features you keep/omit) yields the best accuracy. $\endgroup$ – M. Austin Jan 12 '20 at 1:38
  • $\begingroup$ Hi @M.Austin - would you mind to write it as an elaborate answer. So it can help people like me $\endgroup$ – The Great Jan 12 '20 at 4:49

So here is my answer. This is by no means the best answer. Rather it is just a basic general framework that is used.

When I fit a model I typically like to decide the "best fit" by splitting my data into a training set and a validation set. The training set is used to actually fit the model and the validation set is used to test the model's accuracy. The validation set has absolutely no influence on the model that is fit!

So in your case I would use a random sample of 3,000 (for example) to fit as many models as you wanted to test (e.g. with 6, 7, or 8 features). Then with the remaining 1,712 observations I would create a 2 x 2 table that shows predicted outcome (yes or no) vs. actual outcome (yes or no) and use the model that yields the best accuracy.

It is important to remember that just because a certain model has a higher AUC, or R-squared, or whatever, does not mean it will perform well in your validation data set. Often times if the model is fit too strongly to the training data it does not generalize well outside of that data and is what is called "overfit".

There are lots of books on this stuff, and way smarter people on this subject than myself, but hopefully this is a good start. Good luck!

  • $\begingroup$ Hi, thanks for the response. I mean what if a certain model has higher AUC or F1-score in my test dataset? $\endgroup$ – The Great Jan 13 '20 at 0:15
  • $\begingroup$ I mean out of 3-4 models that I run, I see an improvement in AUC and F1 score for two of my models in test set. But let's say RF with 6 features has 1 point improvement in AUC. earlier with 5 features, it was 83% and now it is 84%. whereas SVM has 0.3 improvement. But rest of the models doesn't show any improvement $\endgroup$ – The Great Jan 13 '20 at 0:34
  • $\begingroup$ I see what you are asking (sorry I missed your part above about 10 fold CV!). If you have a specific evaluation criteria in mind then go with the highest, provided the inclusion/exclusion of a certain feature makes sense in the context of the problem. From your output I not see glaring differences (like one AUC = 90% and one is 40%). $\endgroup$ – M. Austin Jan 13 '20 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.