Is there any classifer that can natively support unbalanced datasets? Or what best practices you can suggest to handle such datasets?

For example I want to solve task called "pedestrian detection" classical approach use linear SVM, but it can't handle unbalanced dataset (lots of background examples, small number of positive examples - people).Maybe there is something better than SVM? (I already know about undersampling/oversampling and weighted SVM).

It would be great if in answer you link to some scikit-learn classification algorithm.


Most classifiers in sklearn support unbalanced datasets, through the sample_weight parameter in the clf.fit methods. If you need to fit unbalanced data with a classifier that does not support this option, you can use sampling with replacement to enlarge the smaller class to match the larger one.

Here is an adapted version of the sklearn SVM example demonstrating the sample_weight approach:

import numpy as np
import pylab as pl
from sklearn import svm

X = np.r_[2*np.random.randn(20, 2) - [2, 2], 2*np.random.randn(200, 2) + [2, 2]]
Y = [0] * 20 + [1] * 200
wt = [1/20.]*20 + [1/200.]*200

# fit the model
clf = svm.SVC(kernel='linear')
clf.fit(X, Y, sample_weight=wt)

This question about unbalanced classification using RandomForestClassifier has some additional details.

  • 1
    $\begingroup$ sample_weight is for particular data points, has nothing to do with unbalanced dataset. class_weight is the one. $\endgroup$
    – renakre
    Jun 29 '17 at 12:01

Linear SVM can handle unbalanced data sets just fine by using class-weights on the misclassification penalty. This functionality is available in any decent SVM implementation.

The objective function for class-weighted SVM is as follows:

$$\min_{\xi,\mathbf{w}} \frac{1}{2}\|\mathbf{w}\|^2 + C_{\mathcal{P}}\sum_{i\in\mathcal{P}} xi_i + C_\mathcal{N} \sum_{i\in\mathcal{N}} \xi_i, $$

where the minority class uses a higher misclassification penalty. A common heuristic is as follows: $$C_\mathcal{P} \times |\mathcal{P}| = C_\mathcal{N} \times |\mathcal{N}|,$$ with $|\mathcal{P}|$ and $\mathcal{N}|$ the number of positive and negative training samples, respectively.

  • $\begingroup$ Why not just use a bias term (aka intercept from GLMs)? $\endgroup$
    – purple51
    May 7 '15 at 6:51
  • $\begingroup$ @purple51 changing the intercept implies translating the separating hyperplane, which is not necessarily optimal. Modifying misclassification costs between classes also enables rotating the separating hyperplane, if necessary. $\endgroup$ May 7 '15 at 6:54
  • $\begingroup$ Right, although I suspect that optimizing the class weights is likely to lead to more severe overfitting than a simple intercept, especially when there's a small number of positives to begin with. $\endgroup$
    – purple51
    May 7 '15 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.