0
$\begingroup$

I have used a "adabag"(boosting + bagging) model on imbalanced dataset (6% positive), I have tried to maximized the sensitivity while keeping the accuracy above 70% and the best result I got where: ROC= 0.711 SENS=0.94 SPEC=0.21

the results aren't Inhofe especially the bad specificity. any suggestion on how to improve the result? optimization solution? penalty function?

this is the code:

ctrl <- trainControl(method = "cv",
                     number = 5,
                     repeats = 2, 
                     p = 0.80,
                     search = "grid", 
                     initialWindow = NULL, 
                     horizon = 1,
                     fixedWindow = TRUE,
                     skip = 0,
                     verboseIter = FALSE,
                     returnData = TRUE,
                     returnResamp = "final",
                     savePredictions = "all",
                     classProbs = TRUE,
                     summaryFunction = twoClassSummary,
                     preProcOptions = list(thresh = 0.80, ICAcomp = 3, k = 7, freqCut = 90/10,uniqueCut = 10, cutoff = 0.2),
                     sampling = "smote",
                     selectionFunction = "best",
                     index = NULL,
                     indexOut = NULL,
                     indexFinal = NULL,
                     timingSamps = 0,
                     predictionBounds = rep(FALSE, 2),
                     seeds = NA,
                     adaptive = list(min = 5,alpha = 0.05, method = "gls", complete = TRUE),
                     trim = FALSE,
                     allowParallel = TRUE)

grid <- expand.grid(maxdepth = 25, mfinal = 4000)

classifier <- train(x = training_set[,-1],y = training_set[,1], method = 'AdaBag',trControl = ctrl,metric = "ROC",tuneGrid = grid)
prediction <- predict(classifier, newdata= test_set,'prob')

plot from classifierplots package:

enter image description here

update:

as suggested I tried xgboost.

here is the code:

gbmGrid <- expand.grid(nrounds = 50, eta = 0.3,max_depth = 3,gamma = 0,colsample_bytree=0.6,min_child_weight=1,subsample=0.75)

ctrl <- trainControl(method = "cv",
                     number = 10,
                     repeats = 2, 
                     p = 0.80,
                     search = "grid", 
                     initialWindow = NULL, 
                     horizon = 1,
                     fixedWindow = TRUE,
                     skip = 0,
                     verboseIter = FALSE,
                     returnData = TRUE,
                     returnResamp = "final",
                     savePredictions = "all",
                     classProbs = TRUE,
                     summaryFunction = twoClassSummary,
                     sampling = "smote",
                     selectionFunction = "best",
                     index = NULL,
                     indexOut = NULL,
                     indexFinal = NULL,
                     timingSamps = 0,
                     predictionBounds = rep(FALSE, 2),
                     seeds = NA,
                     adaptive = list(min = 5,alpha = 0.05, method = "gls", complete = TRUE),
                     trim = FALSE,
                     allowParallel = TRUE)

classifier <- train(x = training_set[,-1],y = training_set[,1], method = 'xgbTree',metric = "ROC",trControl = ctrl,tuneGrid = gbmGrid)
prediction <- predict(classifier, newdata= test_set[,-1],'prob')

plot from classifierplots package:

enter image description here

$\endgroup$
  • 2
    $\begingroup$ Don't optimize for accuracy with an imbalanced set. You can create a 94% accurate classifier by just trivially labelling everything as negative. $\endgroup$ – Calimo Mar 26 '17 at 19:11
  • $\begingroup$ @Calimo I know that, is something in my code optimizing the accuracy?? $\endgroup$ – HilaD Mar 26 '17 at 19:34
  • $\begingroup$ 1) AUC 0.69 sometimes is all you can get; 2) I would try either an xgboost as @AaronDefazio suggested or a glmnet (much simpler!); 3) Sampling is not always a good way, from my exp. using raw data gives better results; 4) Be careful with pre-processing; 5) maxdepth = 25 is a lot, have you tried 2 or 3 first? $\endgroup$ – m-dz Mar 29 '17 at 13:30
  • $\begingroup$ @m-dz what do you mean by be careful with pre-processing? I have 61 features that's why max depth is high, less then that I get worse result. I know its a lot but I cant really reduced them because of the imbalanced problem. $\endgroup$ – HilaD Mar 29 '17 at 13:35
  • $\begingroup$ "Too much pre-processing" can harm the results, e.g. you are using the nearZeroVar() which can remove important variables. Try AdaBoost and xgboost (here you need to recode categorical variables to dummy ones) with depth 2 or 3, they worked well for 1% imbalance. $\endgroup$ – m-dz Mar 29 '17 at 13:44
1
$\begingroup$

There is a vast literature on methods for handling class imbalance. The Caret library you are using has methods for dealing with class imbalance, documented here. It looks like you are already using the SMOTE method as specified by the sampling parameter in your code snippet currently, which is a very reasonable choice. You are unlikely to get an improvement using other methods.

I think you'll find in practice that using the whole dataset as-is will give you the best results. You're more likely to get an improvement using an alternative ML algorithm or implementation. SMOTE and related methods are more for use with classical predictors such as least squares or logistic regression, with flexible methods such as boosting they rarely help.

In terms of alternative algorithms, I would recommend xgboost, which is closely related to AdaBag but often superior in practice.

If you post diagnostic plots (such as from the classifierplots package) we may be able to provide more specific advice.

EDIT: looking at the diagnostic plots, nothing stands out to me as a huge problem. The calibration curve is poor, but that's typical when using boosting methods. AUC values in the high 0.6 to low 0.8 is typical for real-world problems where noise in the label is involved. Is your label expected to be noisy? It's only for pattern recognition type problems where you can really expect to get higher AUC values.

$\endgroup$
  • $\begingroup$ I didn't know about classifierplots package and it's just great. thank you for that. I have edited my post and add the diagnostic plot. I will try xgboost and I mostly know all the methoud for dealing with imbalanced data set, I just don't know how to improve the results. $\endgroup$ – HilaD Mar 29 '17 at 12:56
  • 1
    $\begingroup$ +1 for the classifierplots and xgboost. $\endgroup$ – m-dz Mar 29 '17 at 13:24
  • $\begingroup$ update: tried xgboost in caret and got roc :0.677924 sensitivity: 0.8842105 specificity: 0.225. still not good, any suggestions? $\endgroup$ – HilaD Apr 2 '17 at 8:47
  • $\begingroup$ There are many other algorithms and feature preprocessing approaches you could use, but you are essentially hitting diminishing returns in terms of your time invested in improving the results. Additionally, with the test set size you're using you won't be able to distinguish between small improvements in AUC, the confidence interval for AUC is 0.60-0.77 in the plots you included. Is there any structure in your problem that could be used? You haven't given us any information on the problem yet. $\endgroup$ – AaronDefazio Apr 2 '17 at 13:05
  • $\begingroup$ I didn't understand this Q: "Is there any structure in your problem that could be used? ". the problem is predicting readmission in ICU. $\endgroup$ – HilaD Apr 4 '17 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.