1
$\begingroup$

I have a two-class classification problem with very unbalanced data (~1:1000 Yes/No ratio). The initial model class I'd like to try is regular glm. So there are two issues need to be addressed: 1) feature selection 2) unbalanced data

Below is the approach I've taken:

1) First I sliced the dataset into training and testing.

2) Then I used the rfe function on the training set to conduct feature selection using ROC as metric.

glmProfile<-rfe(data.frame(x4),y,sizes = subsets, rfeControl = ctrl,trControl =trainctrl,metric = "ROC")

It works very well.

3) To deal with the unbalanced data issue, I then used the roc function in pROC package to find the optimal probability threshold on the ROC curve (using testing data).

So the questions I have are:

1) Is this approach acceptable? using the testing set to find optimal probability threshold seems not ideal. I read that we can either slice out another portion of data for this purpose (I don't have a lot of True cases so maybe cannot afford this approach) or use customized function in caret (train) to find optimal probability threshold using resampling (http://topepo.github.io/caret/custom_models.html#Illustration5), but the example given is for random forest model. Is it possible to implement this procedure with glm?

2) the rfe procedure is based on ROC as metric, so I assume it's ok even if the default probability threshold is 0.5 in rfe?

Thank you!

$\endgroup$
  • $\begingroup$ Do you have a full working example for the second step? -> 2) Then I used the rfe function on the training set to conduct feature selection using ROC as metric. $\endgroup$ – diugalde Jan 15 '17 at 23:10
  • $\begingroup$ @diugalde. See an example here machinelearningmastery.com/… $\endgroup$ – ybeybe Jan 17 '17 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.