11
$\begingroup$

I have a very imbalanced big dataset (500000 instances, 60 features) which is prone to changes (increase in size and number of features). But what will stay fixed is the imbalance in the classes, this is, class 0 will always be the dominant one. On average, 90% of the data will be in class 0, and the rest 10% in class 1.

I am interested in classifying as accurately as possible instances with class label 1, so I want to increase its cost of misclassification.

The classifier I chose is RandomForest and in order to account for the class imbalance I am trying to adjust the weights, then evaluate using StratifiedKFold and then plotting the corresponding roc_curve for respective the k fold.

This is the code for my classifier:

 clf1 = RandomForestClassifier(n_estimators=25, min_samples_leaf=10, min_samples_split=10,
      class_weight = "balanced", random_state=1, oob_score=True)

 sample_weights = array([9 if i == 1 else 1 for i in y])

I looked through the documentation and there are some things I don't understand. I tested all these methods but the difference in the evaluation metrics was minimal so I have a hard time identifying which settings optimize my classifier.

Needless to say even though I use weighting the prediction power of my model is very low, with sensitiviy being on average 0.2

These are my questions:

  • should sample_weight and class_weight be used together simultaneously?
  • between class_weights = "balanced" and class_weights = balanced_subsamples which is supposed to give a better performance of the classifier
  • is sample_weight supposed to be adjusted always according to ratio of imbalance in the samples?
  • class_weights = balanced_subsamplesand sample_weight give an execution error when used simultaneously. why?

Also if there is a better approach for evaluating the classifier please do let me know.

$\endgroup$
3
  • 4
    $\begingroup$ You are using the sample_weights wrong. What you want to use is the class_weights. Sample weights are used to increase the importance of a single data-point (let's say, some of your data is more trustworthy, then they receive a higher weight). So: The sample weights exist to change the importance of data-points whereas the class weights change the weights to correct class imbalance. They can be used together with their purpose in mind. In your example, using class weights has no effect whatsoever, because you abused the sample weights to do the job of the class weights. $\endgroup$ – Mayou36 Nov 7 '16 at 10:34
  • $\begingroup$ To add to @Mayou36's comment, class_weight are passed on to sample_weight as well. So be careful while using them. $\endgroup$ – Garima Jain Mar 14 '19 at 16:22
  • $\begingroup$ @GarimaJain just to be explicit about your (true) comment: the weight of datapoint i is given by class_weight[class_of_datapoint_i] times sample_weight[i]. $\endgroup$ – Mayou36 Mar 15 '19 at 13:24
5
$\begingroup$

You are using the sample_weights wrong. What you want to use is the class_weights. Sample weights are used to increase the importance of a single data-point (let's say, some of your data is more trustworthy, then they receive a higher weight). So: The sample weights exist to change the importance of data-points whereas the class weights change the weights to correct class imbalance. They can be used together with their purpose in mind. In your example, using class weights has no effect whatsoever, because you abused the sample weights to do the job of the class weights.

$\endgroup$
3
  • $\begingroup$ I've copied this comment by @Mayou36 as a community wiki answer because they are, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions. $\endgroup$ – mkt - Reinstate Monica May 21 '19 at 11:27
  • $\begingroup$ Fair enough. However, not all of the questions in the post have been addressed and it would be nice if more of them are answered. $\endgroup$ – RMS May 28 '19 at 8:27
  • $\begingroup$ @Roxanne FWIW, you might get more answers if you post questions separately (though there are cases when putting some together makes sense). $\endgroup$ – mkt - Reinstate Monica May 28 '19 at 19:58
2
$\begingroup$

Q&A

  1. Should sample_weight and class_weight be used together simultaneously

If your goal is to weight your classes because they are imbalanced, you can use either. Using class_weight="balanced is the same as sample_weight=[n_samples].

I tested it with an unbalanced set in kaggle. I estimated the "sample_weight" based on what was given in the sklearn docs: n_samples / (n_classes * np.bincount(y))

Note: "A guide as to what weights to give is to make them inversely proportional to the class populations"--- People at Berkeley


  1. between class_weights = "balanced" and class_weights = balanced_subsamples which is supposed to give a better performance of the classifier

You can easily test it for your dataset. It's hard for me to say. For me, recall was higher (9%) and precision was lower by 9% (on the minority) dataset.


  1. is sample_weight supposed to be adjusted always according to ratio of imbalance in the samples?

Depends on what you care about. If you want the prediction error (1-recall) of all classes to be the same, then you can look at the different recalls, change your weights and until they balance out. Note: This WILL reduce your overall prediction error.

This is suggested by "some people" at Uni of Berkeley. However, I was not able to balance out my 1-recall despite increasing my minority class weight by a lot (1000 from 26).

Here is a comparison table for one of the kaggle contests I am working on:

tr recall vd recall tr prec vd prec
baseline 37.4 26.9 94.8 79.6
balanced (1,26) 67.6 (80%) 30.2 (12%) 88.7 (-6.5%) 75.3 (-5.5%)
subsample balanced 67.5 (80%) 32.9 (22%) 88.0 (-7.2%) 70.9 (-11.7%)
overweighting (1,1000) 65.3 (80%) 25.1 (-7%) 84.5 (-10.9%) 84.0 (5%)

  1. class_weights = balanced_subsamples and sample_weight give an execution error when used simultaneously. why?

Not sure.


$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.