I am searching for a classification score, preferably provided by Python scikit-learn, to evaluate classification in a cross-validation routine.
This classification score must be suitable for:
- strong class imbalance
- multiclass classification
The cardinality of the classes is the following:
N Class1 19 Class2 34 Class3 8 Class4 17
I defined a custom scorer based on ROC AUC score from sklearn. Basically I extended it to the multi-class problem by averaging the different scores for each class in a one-vs-all fashion. Is this feasible? Are there drawbacks in this approach?
Here is the Python/sklearn code:
from sklearn.metrics import roc_auc_score from sklearn.preprocessing import LabelBinarizer def custom_avg_roc_auc_score(truth, pred): lb = LabelBinarizer() lb.fit(truth) truth = lb.transform(truth) pred = lb.transform(pred) return roc_auc_score(truth, pred, average="macro") avg_roc_auc_scorer = make_scorer(custom_avg_roc_auc_score)