I have a two-class classification problem with very unbalanced data (~1:1000 Yes/No ratio). The initial model class I'd like to try is regular glm. So there are two issues need to be addressed: 1) feature selection 2) unbalanced data
Below is the approach I've taken:
1) First I sliced the dataset into training and testing.
2) Then I used the rfe function on the training set to conduct feature selection using ROC as metric.
glmProfile<-rfe(data.frame(x4),y,sizes = subsets, rfeControl = ctrl,trControl =trainctrl,metric = "ROC")
It works very well.
3) To deal with the unbalanced data issue, I then used the roc function in pROC package to find the optimal probability threshold on the ROC curve (using testing data).
So the questions I have are:
1) Is this approach acceptable? using the testing set to find optimal probability threshold seems not ideal. I read that we can either slice out another portion of data for this purpose (I don't have a lot of True cases so maybe cannot afford this approach) or use customized function in caret (train) to find optimal probability threshold using resampling (http://topepo.github.io/caret/custom_models.html#Illustration5), but the example given is for random forest model. Is it possible to implement this procedure with glm?
2) the rfe procedure is based on ROC as metric, so I assume it's ok even if the default probability threshold is 0.5 in rfe?