I have been working on this project for over a year now and I believe i finally have things figured out. Mainly i'm looking for any suggestions or things i'm doing wrong with my process, but i also have a couple specific questions.

  • Using sci-kit and optunity in python, binary classification

  • Data : 2200 Observations, 110 features, the outputs are approximately 50/50 so the best i can hope for is maybe 55-60% correct classifications.

  • I split the data into a 70% training set, 20% test set, 10% validation set

  • I wrote a feature selection algorithm that on each increased feature number, removes the worst performing x number of features based on time constraints. It ranks features based on the number of times they appeared in sets that performed in the top 30% and throws out the ones that appeared the least. (any thoughts on this being a resonable method would be appreciated) I used PCA to identify the optimum number of features to use in the final model as well as RFECV in scikit. At this point i used the learning model with default hyperparameters. This was all done using CV on the training data.

  • Once I have identified the best performing feature sets using default hyperparameters, i optimize the hyperparameters on the test data, again using CV and particle swarm/nelder-mead algorithms.

  • At this point i assume the model has been trained properly and can test it on the validation data.

This process does work for me, im mainly just looking for improvements. I am using this process with multiple linear/nonlinear learners, trying different scoring metrics, ect..

Question: Is there a better point in the process to optimize the hyperparameters? Should I be redoing feature selection after determining correct hyperparameters?

Question: Is there a better feature selection method than this? This works, but im starting to have a feeling that it might overfit to the training data and looking for other options.

Question: Ive read that ROC AUC is probably the best scoring metric to use for binary classification because it takes into account differences in the number of class labels. In my case, there is pretty much a 50/50 split on class labels, and the only thing that matters in my results is the number of correct predictions. Given the nature of my data, would you recommend using accuracy as the scoring metric or stick with AUC?

Anyway, long post but any help would be greatly appreciated. Assuming i update any mistakes i feel this post could be very useful for alot of people too. Thanks in advance!


1 Answer 1


Regarding your last question: I often find it helpful to compare multiple metrics when evaluating a model (e.g. precision, recall, accuracy). Given that your data is split 50/50 between both classes, both of these options are valid.

If you want to stick to only one, this post argues in favor of AUC-ROC.

  • $\begingroup$ Thanks! Yeah i had come across that post before. I think i'll end up using both metrics to pick my final model. $\endgroup$
    – user79587
    Commented Apr 21, 2016 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.