New to ML,I have used smote/sampsize for the first time, so sorry if the questions are very basic.I have a dataset with a factor response variable ("Y" , "N" )in the ratio(Y:N=3:7)(classification with imbalanced response variable)
sample.split(credit$Risk,SplitRatio=0.70) to split my train and test dataset.
My trainControl & tuneGrid is as follows:
Irrespective of the model(RF/ranger,cart) model seems to overfits ..for example
confusionMatrix(predTrain,train$Risk,positive="Y") Confusion Matrix and Statistics Reference Prediction N Y N 438 3 Y 52 207 Accuracy : 0.9214 95% CI : (0.899, 0.9403) No Information Rate : 0.7 **P-Value [Acc > NIR] : < 2.2e-16** Kappa : 0.8246 Mcnemar's Test P-Value : 9.651e-11 Sensitivity : 0.9857 Specificity : 0.8939 Pos Pred Value : 0.7992 Neg Pred Value : 0.9932 Prevalence : 0.3000 Detection Rate : 0.2957 Detection Prevalence : 0.3700 Balanced Accuracy : 0.9398 'Positive' Class : Y Confusion Matrix and Statistics Reference Prediction N Y N 153 27 Y 57 63 Accuracy : 0.72 95% CI : (0.6655, 0.7701) No Information Rate : 0.7 **P-Value [Acc > NIR] : 0.245581** Kappa : 0.3913 Mcnemar's Test P-Value : 0.001555 Sensitivity : 0.7000 Specificity : 0.7286 Pos Pred Value : 0.5250 Neg Pred Value : 0.8500 Prevalence : 0.3000 Detection Rate : 0.2100 Detection Prevalence : 0.4000 Balanced Accuracy : 0.7143 'Positive' Class : Y
My questions are:
- If I use
sampling="smote"or a Loss in CART, the p-value for test goes beyond acceptable limit . If I don't use these parameters ..p-vale for both train and test is within 5% significance but results are not good ,for instance Sensitivity drops/F score drops etc. Do we need to give importance to model being statistically significant when we are specifically telling model to give higher preference to class with less no of observation ?
- what is sampsize actually? For example if I have 1000 obs and doing a 3 fold cv,then each fold will be approx. 300 and hence sample size would be
c("N"=150,"Y"=150)considering I am trying to balance? Is this understanding right?
- for ratio like 7:3, do I even need to balance using smote or sampsize? The variables I am using show high predictive power(varImp) and intuitively also seem appropriate, so not sure why this overfitting (even reducing the no. of predictor variable to 3 most imp does not help much)