20
$\begingroup$

I ran a 10-fold cross validation on different binary classification algorithms, with the same dataset, and received both Micro- and Macro averaged results. It should be mentioned that this was a multi-label classification problem.

In my case, true negatives and true positives are weighted equally. That means correctly predicting true negatives is equally important as correctly predicting true positives.

The micro-averaged measures are lower than the macro averaged ones. Here are the results of a Neural Network and Support Vector Machine:

enter image description here

I also ran a percentage-split test on the same dataset with another algorithm. The results were:

enter image description here

I would prefer to compare the percentage-split test with the macro-averaged results, but is that fair? I don't believe that the macro-averaged results are biased because true positives and true negatives are weighted equally, but then again, I wonder if this is the same as comparing apples with oranges?

UPDATE

Based on the comments I will show how the micro and macro averages are calculated.

I have 144 labels (the same as features or attributes) that I want to predict. Precision, Recall and F-Measure are calculated for each label.

---------------------------------------------------
LABEL1 | LABEL2 | LABEL3 | LABEL4 | .. | LABEL144
---------------------------------------------------
   ?   |    ?   |    ?   |   ?    | .. |     ?
---------------------------------------------------

Considering a binary evaluation measure B(tp, tn, fp, fn) that is calculated based on the true positives (tp), true negatives (tn), false positives (fp), and false negatives (fn). The macro and micro averages of a specific measure can be calculated as follows:

enter image description here

enter image description here

Using these formulas we can calculate the micro and macro averages as follows:

enter image description here

enter image description here

So, micro-averaged measures add all the tp, fp and fn (for each label), whereafter a new binary evaluation is made. Macro-averaged measures add all the measures (Precision, Recall, or F-Measure) and divide with the number of labels, which is more like an average.

Now, the question is which one to use?

$\endgroup$
  • $\begingroup$ I don't know what you mean by "micro" and "macro" and I doubt I'm the only one $\endgroup$ – shadowtalker Jun 14 '15 at 19:47
  • $\begingroup$ When you ask which to use, what is the intended use? Choosing between the two methods, summarizing results, or something else? $\endgroup$ – Sean Easter Jun 20 '15 at 16:05
  • 1
    $\begingroup$ The intended use is to figure out which model is the most superior, and to tell something about how well it performs. I found out that Micro measurements are superior according to: Forman, George, and Martin Scholz. "Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement." ACM SIGKDD Explorations Newsletter 12.1 (2010): 49-57. $\endgroup$ – Kenci Jun 29 '15 at 21:30
  • $\begingroup$ @Kenci, I belive you should post that as a reply to your own question and confirm it as being the right anwer. Thanks for the reference! $\endgroup$ – fnl Jul 27 '15 at 9:58
  • $\begingroup$ FYI Micro vs weighted F1 score $\endgroup$ – Franck Dernoncourt Dec 28 '16 at 17:08
26
$\begingroup$

If you think all the labels are more or less equally sized (have roughly the same number of instances), use any.

If you think there are labels with more instances than others and if you want to bias your metric towards the most populated ones, use micromedia.

If you think there are labels with more instances than others and if you want to bias your metric toward the least populated ones (or at least you don't want to bias toward the most populated ones), use macromedia.

If the micromedia result is significantly lower than the macromedia one, it means that you have some gross misclassification in the most populated labels, whereas your smaller labels are probably correctly classified. If the macromedia result is significantly lower than the micromedia one, it means your smaller labels are poorly classified, whereas your larger ones are probably correctly classified.

If you're not sure what to do, carry on with the comparisons on both micro- and macroaverage :)

This is a good paper on the subject.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.