I have a binary classification problem with ~12000 samples and 215 features. I ranked the features based on their importance calculated by Random Forest. Then several classifiers have been trained on dataset using 10-fold cross validation. The process of training each classifier started with using the most predominant feature and adding the next important feature until all of the features are used. Therefore, we can determine how many features each classifier needs to obtain its minimum classification error. So for each classifier, we know the classification error, the variance of error (obtained among 10-fold cross validation), and the number of required features to get that minimum error.

I would like to weight each classifier based on these three parameters including: error, variance of error, and number of features (complexity of the model), but not sure what the best way is to integrate these parameters.

For example, how should I rank the following models (NOTE they are not real models):

Model I: err = 0.24, var = 0.03, number of required features = 10

Model II: err = 0.20, var = 0.02, number of required features = 22

Model III: err = 0.19, var = 0.06, number of required features = 17

Your thoughts and comments are much appreciated.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.