I am searching for a classification score, preferably provided by Python scikit-learn, to evaluate classification in a cross-validation routine.
This classification score must be suitable for:
- strong class imbalance
- multiclass classification
The cardinality of the classes is the following:
N
Class1 19
Class2 34
Class3 8
Class4 17
Update
I defined a custom scorer based on ROC AUC score from sklearn. Basically I extended it to the multi-class problem by averaging the different scores for each class in a one-vs-all fashion. Is this feasible? Are there drawbacks in this approach?
Here is the Python/sklearn code:
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import LabelBinarizer
def custom_avg_roc_auc_score(truth, pred):
lb = LabelBinarizer()
lb.fit(truth)
truth = lb.transform(truth)
pred = lb.transform(pred)
return roc_auc_score(truth, pred, average="macro")
avg_roc_auc_scorer = make_scorer(custom_avg_roc_auc_score)