How to compare and evaluate models for a new feature? I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.
When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th & 8th) feature, I see improvement only in one of the models (LR scikit shows improvement in AUC for 7th feature only whereas Xgboost shows improvement when 7th and 8th features are added).
I also understand that AUC may not be the reliable metric, So I can use log loss for logistic regression. But how do I use one metric which can help me compare the performance of all models? Similar to log loss, is there any other metric for other algorithms like Random Forest, Xgboost, SVM etc?
I also see that we have cochran's Q test which can help us know whether classifiers are different or not? But can this help in picking one classifier over another?

Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models? How should I interpret this?
Please note that I split the data into train and test and did 10 fold CV on train data.
So, how do I know that this newly added features are really helping in improving the model performance? Is there any statistic to find this?
Can help me with this?
 A: So here is my answer. This is by no means the best answer. Rather it is just a basic general framework that is used.
When I fit a model I typically like to decide the "best fit" by splitting my data into a training set and a validation set. The training set is used to actually fit the model and the validation set is used to test the model's accuracy. The validation set has absolutely no influence on the model that is fit! 
So in your case I would use a random sample of 3,000 (for example) to fit as many models as you wanted to test (e.g. with 6, 7, or 8 features). Then with the remaining 1,712 observations I would create a 2 x 2 table that shows predicted outcome (yes or no) vs. actual outcome (yes or no) and use the model that yields the best accuracy. 
It is important to remember that just because a certain model has a higher AUC, or R-squared, or whatever, does not mean it will perform well in your validation data set. Often times if the model is fit too strongly to the training data it does not generalize well outside of that data and is what is called "overfit". 
There are lots of books on this stuff, and way smarter people on this subject than myself, but hopefully this is a good start. Good luck!
