0
$\begingroup$

I have 10 imbalanced datasets. Classes are : 1, 2, 3, ..., 10,11,12.

I used as evaluation metrics for my datasets accuracy and F-measure.

The F-messure of each class in each dataset is as below:

enter image description here

Is it wrong to use accuracy and F-measure as evaluation metrics in this case of datasets ? please see results in the following picture :

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

I would strongly suggest against usign accuracy as a measure for model performance when working with unbalanced datasets. If 99% of the test set belongs to class A and my model always predicts class A, it will have a 99% accuracy despite being completely useless.

F-score (I assume F1-score) is fine as it makes a trade-off between precision and recall. If your predictions come with a probability, the area under the ROC curve (AUC) is an interesting measure, but I don't know if that's the case, so I would go for these three: precision, recall and F1-score. Use F1-score as the main reference, but dismiss any model with precision or recall below acceptable levels.

NOTE: Precision is the proportion of predicted positives that are actually positive. Recall is the amount of real positives that are identified as such. Unless there is a reason to do it otherwise, "positive" refers to the minority class

$\endgroup$
1
  • 1
    $\begingroup$ Large imbalance implies that one can only predict tendencies, not individual outcomes. Tendencies = probabilities. And use a proper accuracy scoring rule as discussed here. $\endgroup$ Jun 14, 2019 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.