# Is there any strategy for validating the result of a general comparison between several confusion matrices?

Disclaimer:

Recently we have developed a python library named PyCM specialized for analyzing multi-class confusion matrices. A compare system has been added in version 2 of this module in order to generally (not considering the subject of the problem) compare the resulted confusion matrices from different classification methods over a unique data-set.

Now, we are searching for a strategy which can validate the result of this option.

This strategy can be either a mathematical proof or a counterexample.

For example suggesting two close confusion matrices which had been compared and the comparison is validated can be helpful to answer this question.

P.S.1. In order to find out how this module works please read the Compare section at this document.

Website: http://www.pycm.ir/

P.S.3. The compare option is a combination of several overall and class-based benchmarks. Each of the benchmarks evaluates the performance of the classification algorithm from good to poor and give them a numeric score. The score of good performance is 1 and for the poor performance is 0.

After that, two scores are calculated for each confusion matrices, overall and class-based. The overall score is the average of the score of four overall benchmarks which are Landis & Koch, Fleiss, Altman, and Cicchetti. And with the same manner, the class-based score is the average of the score of three class-based benchmarks which are Positive Likelihood Ratio Interpretation, Discriminant Power Interpretation, and AUC value Interpretation. It should be noticed that if one of the benchmarks returns none for one of the classes, that benchmarks will be eliminated in total averaging. If user set weights for the classes, the averaging over the value of class-based benchmark scores will transform to a weighted average.

If the user sets the value of by_class boolean input True, the best confusion matrix is the one with the maximum class-based score. Otherwise, if a confusion matrix obtains the maximum of both overall and class-based score, that will be the reported as the best confusion matrix but in any other cases, the compare object doesn’t select best confusion matrix.

• I recommend enriching the question a bit more, give more context and examples if possible. The questions is regarding a topic that is likely to get really good answers on this site. You need to clarify a bit though. – Zhubarb Apr 18 '19 at 8:09
• @Berkan I just update the post. Do you think it's clear now? – Alireza Zolanvari Apr 18 '19 at 8:14
• Can you clarify in the post? How are you comparing? What do you mean by validate? – Björn Apr 18 '19 at 8:17
• @Björn The technique of comparing is proposed at the mentioned document (It's a bit long to explain it here). Imagine the situation that you use two different algorithms for a classification problem and as a result, have two different confusion matrices. You need to see which of the algorithms have a better result, so compare the confusion matrices. This module tries to do this comparison and announces the better result. We want to check if the module does this comparison right. – Alireza Zolanvari Apr 18 '19 at 8:23
• The document does not seem very clear and surely there are very different metrics that people would want to optimize for - how could averaging a few metrics "announce" which one is better?! – Björn Apr 18 '19 at 9:19