I am trying to analyze lending club data and want to predict whether a loan is risky or safe using random forest with decision tree as a classifier. The data is imbalanced. It contains one-fourth of Risky Loans and three-fourths of Safe Loan. I did a stratified split of the data into Training and Testing set. Then, I applied SMOTE to the training set and ran 10 fold CV to chose my model. I was getting good sensitivity around 70%. But after I have selected my model and run it on the test set (with 3:1 ratio of class) I got a sensitivity of 15%. How do I improve sensitivity?

  • 2
    $\begingroup$ You stop balancing your data, rebalancing techniques are one of the most misunderstood and overused techniques in machine learning (your data is hardly imbalanced at all, these techniques are intended for situations where a fraction of a percent of your data is the rare class). You predict probabilities with your random forest and then choose a threshold on those probabilities to maximize your decision objective. $\endgroup$ – Matthew Drury Apr 12 '18 at 20:01
  • $\begingroup$ So, how do I choose model complexity? Should I run k fold on train data, choose on sensitivity and then train that model and tweak roc on test to get good sensitivity? $\endgroup$ – KAY_YAK Apr 12 '18 at 23:41
  • $\begingroup$ I am also confused about thresholding probability. For instance, if your chosen threshold is 0.7 and the class probabilities are 0.6 and 0.4 what class do you assign? None at all? $\endgroup$ – KAY_YAK Apr 13 '18 at 14:27
  • $\begingroup$ What do you mean the class probabilities 0.6 and 0.4? A model assigns one probability to each observation. Do you mean the probability of y = 1 is 0.6, and your decision threshold is 0.7? Then the probability does not pass your threshold, so you decide negative. $\endgroup$ – Matthew Drury May 15 '18 at 3:46
  • $\begingroup$ As for "how do I choose model complexity", you use cross validation to estimate the hold out loss for varying complexities, then choose the model that minimizes the testing loss. $\endgroup$ – Matthew Drury May 15 '18 at 3:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.