1
$\begingroup$

I am using R caret::train for creating my model with imbalance data:

                0     1
table      30,991 6,179
percentage    83%   17%

therefore I am using DMwR::SMOTE (I get better result preparing the data first rather than using smote internally in caret::train) to generate a 60:40 proportion from my binary decision variable. After executing train function using Random Forest algorithm I get the following tuning parameters for my model:

> best_ind <- caret::best(x = fit$results, metric = "Dist", maximize = FALSE)
    print(fit$results[best_ind,])

then

mtry threshold       ROC      Sens      Spec      Dist     ROCSD      SensSD     SpecSD     DistSD
13    5 0.6289474 0.8943797 0.8347721 0.7573713 0.2937601 0.0060855 0.009429729 0.01417039 0.01269218

Which is for me a good result in terms of Sensitivity (0.83) and Specificity (0.75).

Now when I try to test my model using a testing set (I made a partition from my original data set, before smote) with the predict function:

predicted <- predict(fit, test)
confusionMatrixResult <- caret::confusionMatrix(predicted, test$CausesReadmit)

I get:

Confusion Matrix and Statistics           

          Reference                       
Prediction   NO  YES                      
       NO  2199  284                      
       YES  900  333                      

               Accuracy : 0.6814          
                 95% CI : (0.6661, 0.6963)
    No Information Rate : 0.834           
    P-Value [Acc > NIR] : 1               

                  Kappa : 0.1781          
 Mcnemar's Test P-Value : <2e-16          

            Sensitivity : 0.7096          
            Specificity : 0.5397          
         Pos Pred Value : 0.8856          
         Neg Pred Value : 0.2701          
             Prevalence : 0.8340          
         Detection Rate : 0.5918          
   Detection Prevalence : 0.6682          
      Balanced Accuracy : 0.6246          

       'Positive' Class : NO

In both cases the parameters (sensitivity and specificity) are worse (especially for the second one: 0.53), than in fit model and now the auc is 57.78% and in the fit model was: 89%. Because the model is predicting using new information (testing set), it is reasonable to expect to have a lower performance than in the fit model. What surprises me is the magnitude of such difference.

For example when I set sampling argument from trainControl to one of its values: smote, down, up. The difference between the spec, sens, from fit model and testing it is not that high. Why for this case is such big difference? Is it something we can explain?

The variable importance of my model (sorted by importance) is the following:

> caret::varImp(fit$finalModel, type=2)

                      var importance
1                    svc1 1554.34084
2        RealLengthOfStay  819.57894
3                     dx1  784.11788
4  dischargediagnosiscode  766.96813
5                     dx3  764.68220
6  admittingdiagnosiscode  762.31901
7                     dx2  761.70058
8         physicalzipcode  679.44161
9           QNXTReferToId  638.57844
10        QNXTReferFromId  631.69746
11              QNXTPCPId  630.73276
12              AvgIncome  572.72575
13            Disposition  557.17779
14               AgeGroup  555.83880
15              IsReadmit  490.89471
16               RateCode  414.70836
17    referralservicecode  312.88713
18                   svc2  297.14956
19             PCPChanges  205.11664
20           AbnormalFlag  162.65556
21                 Acuity  131.38040
22                   Risk  120.39911
23         RatePercentage  117.41145
24                 gender  101.00251
25             LateToFill   97.58323
26                   svc3   96.29999
27       iscasemanagement   92.42275
28        PrimaryLanguage   86.07432
29          QNXTProgramId   47.94606
30             IsPriority   23.93185

Based on the above situation I have the following questions:

  1. Is this kind of result reasonable, if so what would be the explanation? How much deviation can be reasonable on average or based on your experience?

  2. Is the imbalance nature of my data the possible reason of this different result or on contrary the cause would be more related to the quality of my predictor variables (and I need to find better ones), etc.

Thanks in advance,

$\endgroup$
2
  • 1
    $\begingroup$ I'm tempted to say you didn't train your model by cross-validation. $\endgroup$
    – SmallChess
    Apr 13, 2017 at 5:13
  • $\begingroup$ @SmallChess, I set for trainControl: method="repeatedcv" (this is cross validation, isn't it) and also repeats=5 and for train function: metric="Dist", tuneLength="20", ntree="1000". $\endgroup$
    – David Leal
    Apr 13, 2017 at 5:22

1 Answer 1

1
$\begingroup$

If I understand your question, it is absolutely normal that a model will not do as well on unseen data as it did on training data. Some techniques like cross-validation attempt to give a better idea of actual performance, but how any particular model changes from training to testing depends on the model, the algorithm, the data, etc and I can't think of any rules of thumb except that your model will never perform as well in the real world as it did when you conceived it, trained it, or tested it. It's a downhill process and the best you can do is try to minimize the falloff by properly using cross-validation, resampling, etc, and also understanding your data very well (looking for future leaks, etc.)

It also makes sense that using SMOTE first on all of your data will give better results than if you (properly) use it inside of your cross-validation loop. See my answer here. Similarly, using it on all of your cross-folds combined in cross-validation will result in overly optimistic results.

This question is pretty basic, so I imagine that this question will be closed as a duplicate, but I can't really search CV right now. Outside of CV, you can start by looking into overfitting and similar topics.

$\endgroup$
1
  • $\begingroup$ I understand, but what is odd for me is that, when I use sampling='smote' in trainControl (internal smote), I get similar spec, and sens (but not good values, around .55), but just in case of external SMOTE I get more higher values in fitting the model and then a higher difference with respect the confusion matrix. I don't know if it a consecuence of having imbalance data or not good predictor variables. This is my doubt. Perhaps other people faced a similar situation. Thxs. $\endgroup$
    – David Leal
    Apr 13, 2017 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.