difference between sample_weight and class_weight RandomForest Classifier

Question

I have a very imbalanced big dataset (500000 instances, 60 features) which is prone to changes (increase in size and number of features). But what will stay fixed is the imbalance in the classes, this is, class 0 will always be the dominant one. On average, 90% of the data will be in class 0, and the rest 10% in class 1.

I am interested in classifying as accurately as possible instances with class label 1, so I want to increase its cost of misclassification.

The classifier I chose is RandomForest and in order to account for the class imbalance I am trying to adjust the weights, then evaluate using StratifiedKFold and then plotting the corresponding roc_curve for respective the k fold.

This is the code for my classifier:

 clf1 = RandomForestClassifier(n_estimators=25, min_samples_leaf=10, min_samples_split=10,
      class_weight = "balanced", random_state=1, oob_score=True)

 sample_weights = array([9 if i == 1 else 1 for i in y])

I looked through the documentation and there are some things I don't understand. I tested all these methods but the difference in the evaluation metrics was minimal so I have a hard time identifying which settings optimize my classifier.

Needless to say even though I use weighting the prediction power of my model is very low, with sensitiviy being on average 0.2

These are my questions:

• should sample_weight and class_weight be used together simultaneously?
• between class_weights = "balanced" and class_weights = balanced_subsamples which is supposed to give a better performance of the classifier
• is sample_weight supposed to be adjusted always according to ratio of imbalance in the samples?
• class_weights = balanced_subsamplesand sample_weight give an execution error when used simultaneously. why?

Also if there is a better approach for evaluating the classifier please do let me know.

Answer 1

You are using the sample_weights wrong. What you want to use is the class_weights. Sample weights are used to increase the importance of a single data-point (let's say, some of your data is more trustworthy, then they receive a higher weight). So: The sample weights exist to change the importance of data-points whereas the class weights change the weights to correct class imbalance. They can be used together with their purpose in mind. In your example, using class weights has no effect whatsoever, because you abused the sample weights to do the job of the class weights.

Answer 2

Q&A

1. Should sample_weight and class_weight be used together simultaneously

If your goal is to weight your classes because they are imbalanced, you can use either. Using class_weight="balanced is the same as sample_weight=[n_samples].

I tested it with an unbalanced set in kaggle. I estimated the "sample_weight" based on what was given in the sklearn docs: n_samples / (n_classes * np.bincount(y))

Note: "A guide as to what weights to give is to make them inversely proportional to the class populations"--- People at Berkeley

---

2. between class_weights = "balanced" and class_weights = balanced_subsamples which is supposed to give a better performance of the classifier

You can easily test it for your dataset. It's hard for me to say. For me, recall was higher (9%) and precision was lower by 9% (on the minority) dataset.

---

3. is sample_weight supposed to be adjusted always according to ratio of imbalance in the samples?

Depends on what you care about. If you want the prediction error (1-recall) of all classes to be the same, then you can look at the different recalls, change your weights and until they balance out. Note: This WILL reduce your overall prediction error.

This is suggested by "some people" at Uni of Berkeley. However, I was not able to balance out my 1-recall despite increasing my minority class weight by a lot (1000 from 26).

Here is a comparison table for one of the kaggle contests I am working on:

	tr recall	vd recall	tr prec	vd prec
baseline	37.4	26.9	94.8	79.6
balanced (1,26)	67.6 (80%)	30.2 (12%)	88.7 (-6.5%)	75.3 (-5.5%)
subsample balanced	67.5 (80%)	32.9 (22%)	88.0 (-7.2%)	70.9 (-11.7%)
overweighting (1,1000)	65.3 (80%)	25.1 (-7%)	84.5 (-10.9%)	84.0 (5%)

---

4. class_weights = balanced_subsamples and sample_weight give an execution error when used simultaneously. why?

Not sure.

---