Classification models:Overfitting due to sampling issue

Question

New to ML,I have used smote/sampsize for the first time, so sorry if the questions are very basic.I have a dataset with a factor response variable ("Y" , "N" )in the ratio(Y:N=3:7)(classification with imbalanced response variable)

I used sample.split(credit$Risk,SplitRatio=0.70) to split my train and test dataset.

My trainControl & tuneGrid is as follows:

trControl=trainControl(method="cv",num=3,classProbs=TRUE,summaryFunction=prSummary,search="grid",sampling="smote")

fit_rf<-train(Risk~chkngAccnt+creditamount+Duration+installmentrate+YAE+age,data=train,tuneGrid=trgridF,method="rf",trControl=trControl,ntree=500,metric="F",strata=train$Risk)

Irrespective of the model(RF/ranger,cart) model seems to overfits ..for example

confusionMatrix(predTrain,train$Risk,positive="Y")

Confusion Matrix and Statistics

          Reference
Prediction   N   Y
         N 438   3
         Y  52 207

               Accuracy : 0.9214         
                 95% CI : (0.899, 0.9403)
    No Information Rate : 0.7            
    **P-Value [Acc > NIR] : < 2.2e-16**      

                  Kappa : 0.8246         

 Mcnemar's Test P-Value : 9.651e-11      

            Sensitivity : 0.9857         
            Specificity : 0.8939         
         Pos Pred Value : 0.7992         
         Neg Pred Value : 0.9932         
             Prevalence : 0.3000         
         Detection Rate : 0.2957         
   Detection Prevalence : 0.3700         
      Balanced Accuracy : 0.9398         

       'Positive' Class : Y 

Confusion Matrix and Statistics

          Reference
Prediction   N   Y
         N 153  27
         Y  57  63

               Accuracy : 0.72            
                 95% CI : (0.6655, 0.7701)
    No Information Rate : 0.7             
    **P-Value [Acc > NIR] : 0.245581**        

                  Kappa : 0.3913          

 Mcnemar's Test P-Value : 0.001555        

            Sensitivity : 0.7000          
            Specificity : 0.7286          
         Pos Pred Value : 0.5250          
         Neg Pred Value : 0.8500          
             Prevalence : 0.3000          
         Detection Rate : 0.2100          
   Detection Prevalence : 0.4000          
      Balanced Accuracy : 0.7143          

       'Positive' Class : Y       

My questions are:

1. If I use sampling="smote" or a Loss in CART, the p-value for test goes beyond acceptable limit . If I don't use these parameters ..p-vale for both train and test is within 5% significance but results are not good ,for instance Sensitivity drops/F score drops etc. Do we need to give importance to model being statistically significant when we are specifically telling model to give higher preference to class with less no of observation ?
2. what is sampsize actually? For example if I have 1000 obs and doing a 3 fold cv,then each fold will be approx. 300 and hence sample size would be c("N"=200,"Y"=100) or c("N"=150,"Y"=150) considering I am trying to balance? Is this understanding right?
3. for ratio like 7:3, do I even need to balance using smote or sampsize? The variables I am using show high predictive power(varImp) and intuitively also seem appropriate, so not sure why this overfitting (even reducing the no. of predictor variable to 3 most imp does not help much)

Answer 1

General answer:

I would not be using SMOTE for a ratio of 7:3. If you really want to balance your precision/recall, try to set some class weights - they are available in R's implementation of Random Forest.

'sampsize' works that you need to give him a vector in the form of c(300,300), where the order depends on the level of the factor variable you passed as strata. These numbers don't necessarily need to be smaller than your sample size, as it is a bootstrap and you're sampling with replacement (but this is my take, I am not sure).

However, with these rations oversampling will decrease your performance more than it will increase it, as you are adding "fake" points. If you're overfitting RF, try to set a max_depth parameter or better tune the mtry, drop the smote and put some class weights. It should work better!