5
$\begingroup$

I have a very imbalanced big dataset (500000 instances, 60 features) which is prone to changes (increase in size and number of features). But what will stay fixed is the imbalance in the classes, this is, class 0 will always be the dominant one. On average, 90% of the data will be in class 0, and the rest 10% in class 1.

I am interested in classifying as accurately as possible instances with class label 1, so I want to increase its cost of misclassification.

The classifier I chose is RandomForest and in order to account for the class imbalance I am trying to adjust the weights, then evaluate using StratifiedKFold and then plotting the corresponding roc_curve for respective the k fold.

This is the code for my classifier:

 clf1 = RandomForestClassifier(n_estimators=25, min_samples_leaf=10, min_samples_split=10,
      class_weight = "balanced", random_state=1, oob_score=True)

 sample_weights = array([9 if i == 1 else 1 for i in y])

I looked through the documentation and there are some things I don't understand. I tested all these methods but the difference in the evaluation metrics was minimal so I have a hard time identifying which settings optimize my classifier.

Needless to say even though I use weighting the prediction power of my model is very low, with sensitiviy being on average 0.2

These are my questions:

  • should sample_weight and class_weight be used together simultaneously?
  • between class_weights = "balanced" and class_weights = balanced_subsamples which is supposed to give a better performance of the classifier
  • is sample_weight supposed to be adjusted always according to ratio of imbalance in the samples?
  • class_weights = balanced_subsamplesand sample_weight give an execution error when used simultaneously. why?

Also if there is a better approach for evaluating the classifier please do let me know.

$\endgroup$
  • 3
    $\begingroup$ You are using the sample_weights wrong. What you want to use is the class_weights. Sample weights are used to increase the importance of a single data-point (let's say, some of your data is more trustworthy, then they receive a higher weight). So: The sample weights exist to change the importance of data-points whereas the class weights change the weights to correct class imbalance. They can be used together with their purpose in mind. In your example, using class weights has no effect whatsoever, because you abused the sample weights to do the job of the class weights. $\endgroup$ – Mayou36 Nov 7 '16 at 10:34
  • $\begingroup$ To add to @Mayou36's comment, class_weight are passed on to sample_weight as well. So be careful while using them. $\endgroup$ – Garima Jain Mar 14 '19 at 16:22
  • $\begingroup$ @GarimaJain just to be explicit about your (true) comment: the weight of datapoint i is given by class_weight[class_of_datapoint_i] times sample_weight[i]. $\endgroup$ – Mayou36 Mar 15 '19 at 13:24
  • $\begingroup$ guys, no hate pls...but I posted this 3 years ago... $\endgroup$ – Roxanne Mar 15 '19 at 15:48
1
$\begingroup$

You are using the sample_weights wrong. What you want to use is the class_weights. Sample weights are used to increase the importance of a single data-point (let's say, some of your data is more trustworthy, then they receive a higher weight). So: The sample weights exist to change the importance of data-points whereas the class weights change the weights to correct class imbalance. They can be used together with their purpose in mind. In your example, using class weights has no effect whatsoever, because you abused the sample weights to do the job of the class weights.

$\endgroup$
  • $\begingroup$ I've copied this comment by @Mayou36 as a community wiki answer because they are, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions. $\endgroup$ – mkt - Reinstate Monica May 21 '19 at 11:27
  • $\begingroup$ Fair enough. However, not all of the questions in the post have been addressed and it would be nice if more of them are answered. $\endgroup$ – Roxanne May 28 '19 at 8:27
  • $\begingroup$ @Roxanne FWIW, you might get more answers if you post questions separately (though there are cases when putting some together makes sense). $\endgroup$ – mkt - Reinstate Monica May 28 '19 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.