My datasets have two classes A and B. The classes should be treated equally (there is no "active/inactive"). The datasets are unbalanced, sometimes A is more frequent, sometimes B is more frequent. Which performance measure should I use?
Accuracy makes no sense on unbalanced datasets. If I get it right, F-measure and AUC assume that there is a active class: F-measure ignores true negatives as it is the harmonic mean of precision and recall. AUC ignores true negatives and false negatives.
So what performance measure should I use? Is AUC(active=A) + AUC(active=B) / 2 a valid option?
Apparently, I missunderstood how AUC works. It does NOT ignore true negatives and false negatives. The ROC curves look different depending on which class is considered the active one, but AUC(active=A) = AUC(active=B).