I'm trying to optimize the hyperparameters of an SVM. I have an unbalanced data set with more than two classes. In some classes very many samples are included in others very few. Using GridSearchCV, I try to find the optimal hyperparameters and chose f1 (macro) for scoring, because the dataset is unbalanced. Furthermore, I set class_weight = 'balanced' and here I am not sure if this really makes sense or is rather counterproductive

def make_f1_scorer():
    f1 = make_scorer(f1_score, average='macro')
    return f1

scoring = make_f1_scorer()

clf=GridSearchCV(LinearSVC(class_weight='balanced'), param_grid=param_grid,cv=5, scoring=scoring, n_jobs=-1)

Maybe someone can tell me if this approach is right or another better. Thanks in advance for any help


class_weight option penalizes mistakes in classes differently, with a weight inversely proportional to class frequencies, specifically $\frac{n}{kn_i}$, where $n$ is number of samples, $k$ is number of classes and $n_i$ is number of samples in class $i$, which intentionally notifies the algorithm to also account sensibly for minorities. This option may or may not make a noticeable difference in all problems, but, generally speaking, it also shouldn't harm since one of the motivations of this mechanism is indeed fighting dataset imbalances.

Another thing that I've realized is you don't specify any sampling scheme in fold creation. Stratified sampling assures that folds have similar class distribution. Fortunately, in LinearSVC, the following explanation suits your case, and you use StratifiedKFold under the hood:

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

  • $\begingroup$ Thank your for the great help. $\endgroup$ – Code Now Aug 30 '19 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.