I have been working on this project for over a year now and I believe i finally have things figured out. Mainly i'm looking for any suggestions or things i'm doing wrong with my process, but i also have a couple specific questions.
Using sci-kit and optunity in python, binary classification
Data : 2200 Observations, 110 features, the outputs are approximately 50/50 so the best i can hope for is maybe 55-60% correct classifications.
I split the data into a 70% training set, 20% test set, 10% validation set
I wrote a feature selection algorithm that on each increased feature number, removes the worst performing x number of features based on time constraints. It ranks features based on the number of times they appeared in sets that performed in the top 30% and throws out the ones that appeared the least. (any thoughts on this being a resonable method would be appreciated) I used PCA to identify the optimum number of features to use in the final model as well as RFECV in scikit. At this point i used the learning model with default hyperparameters. This was all done using CV on the training data.
Once I have identified the best performing feature sets using default hyperparameters, i optimize the hyperparameters on the test data, again using CV and particle swarm/nelder-mead algorithms.
At this point i assume the model has been trained properly and can test it on the validation data.
This process does work for me, im mainly just looking for improvements. I am using this process with multiple linear/nonlinear learners, trying different scoring metrics, ect..
Question: Is there a better point in the process to optimize the hyperparameters? Should I be redoing feature selection after determining correct hyperparameters?
Question: Is there a better feature selection method than this? This works, but im starting to have a feeling that it might overfit to the training data and looking for other options.
Question: Ive read that ROC AUC is probably the best scoring metric to use for binary classification because it takes into account differences in the number of class labels. In my case, there is pretty much a 50/50 split on class labels, and the only thing that matters in my results is the number of correct predictions. Given the nature of my data, would you recommend using accuracy as the scoring metric or stick with AUC?
Anyway, long post but any help would be greatly appreciated. Assuming i update any mistakes i feel this post could be very useful for alot of people too. Thanks in advance!
