I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.
When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th & 8th) feature, I see improvement only in one of the models (LR scikit shows improvement in AUC for 7th feature only whereas Xgboost shows improvement when 7th and 8th features are added).
I also understand that
AUC may not be the reliable metric, So I can use
log loss for logistic regression. But how do I use one metric which can help me compare the performance of all models? Similar to
log loss, is there any other metric for other algorithms like
Random Forest, Xgboost, SVM etc?
I also see that we have
cochran's Q test which can help us know whether classifiers are different or not? But can this help in picking one classifier over another?
Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models? How should I interpret this?
Please note that I split the data into train and test and did 10 fold CV on train data.
So, how do I know that this newly added features are really helping in improving the model performance? Is there any statistic to find this?
Can help me with this?