First of all, let me say I am new to Machine Learning and am eager for any sort of feedback. I am attempting to create a predictive attrition model, and my training and test data each have ~ 17% records for leaving. My goal would be to produce a "good" model for predicting attrition.

  1. Is this considered imbalanced? Do I need to use SMOTE or some sort of resampling here?
  2. If yes, I know that in order to use SMOTE, I must do it to the training set only, but I am unsure of how to re-sample while simultaneously cross-validating?
  3. Once the model is constructed, what are "acceptable" levels for accuracy/precision/recall? Should I care about other things as well?

Here is the code I have used for SMOTE:

# Create training and test data sets
# Ensure results are repeatable
training <- createDataPartition(t_final2$attrit,times = 1,p=0.5) %>% unlist()
train_data <- t_final2[training,]
test_data <- t_final2[-training,]

# use SMOTE to artificially upsample attrits in our data set
train_data %<>% as.data.frame()
train_data <- SMOTE(attrit ~., train_data,k=10,perc.over= 100,perc.under = 350)
  0    1 
3227 1844

Next, I used a XGBoost model:

train <- setDT(train_data)
test <- setDT(test_data)

labels <- train$attrit
labels <- as.numeric(labels)-1
ts_label <- test$attrit
ts_label <- as.numeric(ts_label)-1

new_tr <- model.matrix(~.+0,data=train[,-c("attrit"),with=F])
new_ts <- model.matrix(~.+0,data=test[,-c("attrit"),with=F])

dtrain <- xgb.DMatrix(data=new_tr,label=labels)
dtest <- xgb.DMatrix(data=new_ts,label=ts_label)

# Set Parameters
params <- list(booster = "gbtree", objective = "binary:logistic", eta=0.3, gamma=0, max_depth=6, min_child_weight=1, subsample=1, colsample_bytree=1)

# use cv to tune model, and return cv error
xgbcv <- xgb.cv(params = params, data = dtrain, nrounds = 150, nfold = 5, showsd = T, stratified = T, print_every_n = 10, early_stopping_rounds = 10, maximize = F)

# Result
Stopping. Best iteration:
[63]    train-error:0.080457+0.005289   test-error:0.174126+0.018541

# Model Training
xgb1 <- xgb.train (params = params, data = dtrain, nrounds = 63, watchlist = list(val=dtest,train=dtrain), print_every_n = 10, early_stop_round = 10, maximize = F,eval_metric = "error")
xgbpred <- predict(xgb1,dtest)
xgbpred <- ifelse(xgbpred > 0.5,1,0)

# Look at Confusion Matrix
confusionMatrix(xgbpred,ts_label,positive = '1')

Results are in screenshot below. I know that my recall is low, I was hoping to improve the model. Any feedback is appreciated!

Initial Model Results

  • 1
    $\begingroup$ Your largest error is assuming that a 0.5 threshold is appropriate for your problem. Please instead threshold your probabilistic predictions at a place that balances the true and false positive rates in a way that is appropriate for your application. This makes class balance a non issue. $\endgroup$ – Matthew Drury Nov 3 '17 at 1:02
  • $\begingroup$ @Matthew, what would be appropriate true and false positive rates for this application? I do not have enough ML context to know what should be the correct rates in a model like this.That makes sense on class balance. $\endgroup$ – kkann47 Nov 3 '17 at 1:18
  • 1
    $\begingroup$ That depends on what you intend to use the model for. Its possible that you dont even need to make class assignments at all. If you are trying to prioritize investigations into fraud, for example, you would simply want to prioritize cases based on the predicted probability of fraud. To evaluate, you could hypothosise how many a team could investigate in a day, and then determine the teams efficiency if prioritizing based on your models predictions. How you evaluate a decision procedure depends on context. $\endgroup$ – Matthew Drury Nov 3 '17 at 1:22
  • $\begingroup$ Excellent! Thank you I agree that a probability would be more useful in this context. I plan to use this model for headcount planning primarily. So would you try to shoot for true/false positive and rates at around 70%? $\endgroup$ – kkann47 Nov 3 '17 at 1:47
  • $\begingroup$ No, if you're shooting to estimate expected headcount, then hard classification is just not a relevant measure of performance. Your main form of evaluation should be to make sure the model is estimating probabilities accurately. You should make lots of diagnostic plots where you bucket your data together by the various variables in your model, and by the predicted values, and make sure the average predicted probabilities match the average value of $y$. Once you are confident, you can use the model to estimate your expected head count. $\endgroup$ – Matthew Drury Nov 3 '17 at 1:50
  1. It depends on who you ask. If people have a background in fraud detection or something, they would consider < 1% to be imbalanced. But I have seen papers that consider 17% somewhat imbalanced (see Table V here: http://citeseerx.ist.psu.edu/viewdoc/download?doi= I would say it isn't what most people would consider a class imbalance problem; but, in the end, the data don't really care what you label the size of the minority class. I would try out SMOTE, upsampling, and downsampling. Those tend to appear the most-mentioned in reviews of class imbalance. The unbalanced package in R provides a number of good functions (https://cran.r-project.org/web/packages/unbalanced/unbalanced.pdf). Of course, it would be helpful to compare models when you did no sampling techniques so that you could deem if they were really necessary.

  2. For cross-validation, you would do the sampling technique on the training set and not on the test set. If you did this using 5-fold CV, imagine you then have folds labeled A, B, C, D, and E. First, you would do SMOTE on A - D, train the model on A - D, test on E. Second, you would do SMOTE on A - C and E, train on A - C and E, then test on D. Each time, you would save whatever scores you want. There is definitely a way to code this in a clever way that could be parallelized (perhaps a package, like caret, has already done something like this?), but that's how I see CV working in this context.

  3. If you are concerned about class imbalance, then accuracy really shouldn't be much of a concern to you. I take it that you are particularly interested in predicting those 17% of cases, right? If you have a specific cost in mind (such that one of those 17% is 5 times as important as the majority class), then you could change a cost matrix. Otherwise, you could look at precision, recall, F1 score, and perhaps AUC(ROC).

Some references to get you started:

| cite | improve this answer | |
  • $\begingroup$ Thank you Mark, after further research, I believe you are correct in that I will not need SMOTE (or another form of resampling) to answer this question effectively. Yes the 17% is all I care about for the most part, though I am not sure if I can quantify how much more. $\endgroup$ – kkann47 Nov 3 '17 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.