# Classification models:Overfitting due to sampling issue

New to ML,I have used smote/sampsize for the first time, so sorry if the questions are very basic.I have a dataset with a factor response variable ("Y" , "N" )in the ratio(Y:N=3:7)(classification with imbalanced response variable)

I used sample.split(credit$Risk,SplitRatio=0.70) to split my train and test dataset. My trainControl & tuneGrid is as follows: trControl=trainControl(method="cv",num=3,classProbs=TRUE,summaryFunction=prSummary,search="grid",sampling="smote") fit_rf<-train(Risk~chkngAccnt+creditamount+Duration+installmentrate+YAE+age,data=train,tuneGrid=trgridF,method="rf",trControl=trControl,ntree=500,metric="F",strata=train$Risk)

Irrespective of the model(RF/ranger,cart) model seems to overfits ..for example

confusionMatrix(predTrain,train\$Risk,positive="Y")

Confusion Matrix and Statistics

Reference
Prediction   N   Y
N 438   3
Y  52 207

Accuracy : 0.9214
95% CI : (0.899, 0.9403)
No Information Rate : 0.7
**P-Value [Acc > NIR] : < 2.2e-16**

Kappa : 0.8246

Mcnemar's Test P-Value : 9.651e-11

Sensitivity : 0.9857
Specificity : 0.8939
Pos Pred Value : 0.7992
Neg Pred Value : 0.9932
Prevalence : 0.3000
Detection Rate : 0.2957
Detection Prevalence : 0.3700
Balanced Accuracy : 0.9398

'Positive' Class : Y

Confusion Matrix and Statistics

Reference
Prediction   N   Y
N 153  27
Y  57  63

Accuracy : 0.72
95% CI : (0.6655, 0.7701)
No Information Rate : 0.7
**P-Value [Acc > NIR] : 0.245581**

Kappa : 0.3913

Mcnemar's Test P-Value : 0.001555

Sensitivity : 0.7000
Specificity : 0.7286
Pos Pred Value : 0.5250
Neg Pred Value : 0.8500
Prevalence : 0.3000
Detection Rate : 0.2100
Detection Prevalence : 0.4000
Balanced Accuracy : 0.7143

'Positive' Class : Y


My questions are:

1. If I use sampling="smote" or a Loss in CART, the p-value for test goes beyond acceptable limit . If I don't use these parameters ..p-vale for both train and test is within 5% significance but results are not good ,for instance Sensitivity drops/F score drops etc. Do we need to give importance to model being statistically significant when we are specifically telling model to give higher preference to class with less no of observation ?
2. what is sampsize actually? For example if I have 1000 obs and doing a 3 fold cv,then each fold will be approx. 300 and hence sample size would be c("N"=200,"Y"=100) or c("N"=150,"Y"=150) considering I am trying to balance? Is this understanding right?
3. for ratio like 7:3, do I even need to balance using smote or sampsize? The variables I am using show high predictive power(varImp) and intuitively also seem appropriate, so not sure why this overfitting (even reducing the no. of predictor variable to 3 most imp does not help much)
• I dont really understand what parameter are you tuning with your grid. Could you clarify please? – Davide ND Jan 31 at 10:36

'sampsize' works that you need to give him a vector in the form of c(300,300), where the order depends on the level of the factor variable you passed as strata. These numbers don't necessarily need to be smaller than your sample size, as it is a bootstrap and you're sampling with replacement (but this is my take, I am not sure).
However, with these rations oversampling will decrease your performance more than it will increase it, as you are adding "fake" points. If you're overfitting RF, try to set a max_depth parameter or better tune the mtry, drop the smote and put some class weights. It should work better!