8
$\begingroup$

I have a dataset where yes=77 and no=16000, a highly imbalanced dataset. My plan was to identify the most important variables influencing the response variable using random forest and then develop a logistic regression model using the selected variable.

I am planning to use random forest package and resampling technique. I know it will reduce the type II error but increase the type I error. Is Random forest a reasonable technique for analyzing this data? Is there any other machine learning technique which would be more suitable for my case.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

There are usually two methods to deal with imbalanced data while using the random forest model. One approach is cost-sensitive learning and the other is sampling. For extremely imbalanced data, random forest generally tends to be biased towards the majority class.

The cost-sensitive approach would be to assign different weights to different classes. So if the minority class is assigned a higher weight and thus higher misclassification cost, then that can help reduce its biasness towards the majority class. You can use the class weight parameter of random forest in scikit-learn to assign weights to each class.

Secondly, there are different methods of sampling such as oversampling the minority class or undersampling the majority class etc... Although simple sampling methods improve the overall model performance, its preferable to go for a more specialized sampling method such as SMOTE and others to get a better model.

Most of the machine learning models suffer from the imbalanced data problem although there are some reasons to believe that generative models generally tend to perform better in case of imbalanced datasets.

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ @Mathew Drury I've seen your questions and I never answered because I don't have an answer. However, I do have one thought. If one has the/a 'correct' set of variables, then one would expect the data space to be robust in the sense that small perturbations of the data shouldn't change the class label. Of course one has no guarantee that such is the case. One might consider checking robustness using knn. If knn showed that some nhd of a given data point was largely (mostly, entirely?) the same class label, then using smote should be effective. I believe there is a (continued) $\endgroup$
    – meh
    Apr 16, 2018 at 16:56
  • 1
    $\begingroup$ package in R that implements this. Of course this is absolutely not a theoretical explanation. I could conceive of a result that set that based on some measure of robustness as above, using smote would improve accuracy. $\endgroup$
    – meh
    Apr 16, 2018 at 16:58
  • 1
    $\begingroup$ Can you give reference where I can learn more about your generative models statement? $\endgroup$
    – EngrStudent
    Apr 17, 2018 at 16:18
  • 2
    $\begingroup$ I have another question: Given, I have 68 predictors: should I select the most important variable first using Gini index than build prediction model? I am not sure why, my models using no resampling and Smote giving the same result Accuracy: 0.99 and sensitivity: 0.96 $\endgroup$
    – MSilvy
    Apr 17, 2018 at 18:26
  • 1
    $\begingroup$ A few pointers: While selecting features better do it based on CV score. A simple but commonly used method is Leave-one-out CV. Check your feature importance scores based on the trained random forest model. Take the least important features and perform LOO-CV. Which basically needs you to drop each feature and perform CV to check whether removing that feature improves your score or not. You can google it if you are unclear about LOO-CV. $\endgroup$
    – Satwik Bhattamishra
    Apr 17, 2018 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.