First of all, let me say I am new to Machine Learning and am eager for any sort of feedback. I am attempting to create a predictive attrition model, and my training and test data each have ~ 17% records for leaving. My goal would be to produce a "good" model for predicting attrition.
- Is this considered imbalanced? Do I need to use SMOTE or some sort of resampling here?
- If yes, I know that in order to use SMOTE, I must do it to the training set only, but I am unsure of how to re-sample while simultaneously cross-validating?
- Once the model is constructed, what are "acceptable" levels for accuracy/precision/recall? Should I care about other things as well?
Here is the code I have used for SMOTE:
# Create training and test data sets # Ensure results are repeatable set.seed(7) training <- createDataPartition(t_final2$attrit,times = 1,p=0.5) %>% unlist() train_data <- t_final2[training,] test_data <- t_final2[-training,] # use SMOTE to artificially upsample attrits in our data set train_data %<>% as.data.frame() train_data <- SMOTE(attrit ~., train_data,k=10,perc.over= 100,perc.under = 350) table(train_data$attrit) 0 1 3227 1844
Next, I used a XGBoost model:
train <- setDT(train_data) test <- setDT(test_data) labels <- train$attrit labels <- as.numeric(labels)-1 ts_label <- test$attrit ts_label <- as.numeric(ts_label)-1 new_tr <- model.matrix(~.+0,data=train[,-c("attrit"),with=F]) new_ts <- model.matrix(~.+0,data=test[,-c("attrit"),with=F]) dtrain <- xgb.DMatrix(data=new_tr,label=labels) dtest <- xgb.DMatrix(data=new_ts,label=ts_label) # Set Parameters params <- list(booster = "gbtree", objective = "binary:logistic", eta=0.3, gamma=0, max_depth=6, min_child_weight=1, subsample=1, colsample_bytree=1) # use cv to tune model, and return cv error set.seed(7) xgbcv <- xgb.cv(params = params, data = dtrain, nrounds = 150, nfold = 5, showsd = T, stratified = T, print_every_n = 10, early_stopping_rounds = 10, maximize = F) # Result Stopping. Best iteration:  train-error:0.080457+0.005289 test-error:0.174126+0.018541 # Model Training xgb1 <- xgb.train (params = params, data = dtrain, nrounds = 63, watchlist = list(val=dtest,train=dtrain), print_every_n = 10, early_stop_round = 10, maximize = F,eval_metric = "error") xgbpred <- predict(xgb1,dtest) xgbpred <- ifelse(xgbpred > 0.5,1,0) # Look at Confusion Matrix confusionMatrix(xgbpred,ts_label,positive = '1')
Results are in screenshot below. I know that my recall is low, I was hoping to improve the model. Any feedback is appreciated!