Transforming confusion matrix to distance matrix

Question

I have an annotation task which I need to analyze for accuracy, category fine tuning and reliability , there are many annotators assigning multiple set valued categories to items. And I have come up with confusion matrices for each pair of annotators like below:

+-----------------------------------------+
|  |5|  .   1   .   .   .   .   2   .   . |
|   7  |.|  .   .   .   .   .   5   .   1 |
|  34   .  |.|  .   .   .   1   4   .   1 |
|   7   1   .  |.|  .   .   1  46   .   . |
|   .   .   .   .  |.|  1   .   .   .   . |
|   5   .   .   .   .  |.|  .   1   .   . |
|   5   .   .   .   .   .  |.|  .   .   . |
|   5   .   .   .   .   .   1  |2|  .   5 |
| 198   .   1   .   .   2   1  11  |.|  1 |
|   1   .   .   .   .   .   2   .   .  |.||
------------+-----------------------------+
 

To transform it into an undirected distance matrix , I transposed the above and averaged it to get the following symmetric matrix which I need to represent as distance matrix:

|   |5|   3   17    3    .    2    2    3   99    0 |
|    3   |.|   .    0    .    .    .    2    .    0 |
|   17    .   |.|   .    .    .    0    2    0    0 |
|    3    0    .   |.|   .    .    0   23    .    . |
|    .    .    .    .   |.|   0    .    .    .    . |
|    2    .    .    .    0   |.|   .    0    1    . |
|    2    .    0    0    .    .   |.|   0    0    1 |
|    3    2    2   23    .    0    0   |2|   5    2 |
|   99    .    0    .    .    1    0    5   |.|   0 |
|    0    0    0    .    .    .    1    2    0   |.||

I have about 190 such matrices from multiple annotator pairs and there is nothing like reference and test categories as the annotator category assignment needs to be analyzed for fine tuning, merging the categories.

So is the above approach correct to start with and how to transform multiple such matrices (dimensions)into a single visualization allowing to analyze the categories and inter annotator reliability apart from PCA and the usual kappa and related statistics?

Also would such a distance matrix give a correct indication of the real components in PCA for cluster analysis to base further decisions on?

Answer 1

First, in this case, confussion matrix roughly transforms to something like similarity matrix, not distance matrix. If two annotators are often confusing two labels, then it means that these labels are somewhat similar, not distant.

The simplest approach to take account of all such matrices would be to simply calculate their mean. If you are already taking a mean between each pair, then taking a mean of all annotators would not bring much more harm to the information then it is already done.

Second, I do not really see any correlation between such method and PCA (but I may be simply missing something here). Maybe you were thinking about applying the PCA on the data? Either way, does not seem as a good idea as PCA will result in linear combinations of your categories, rather then just "connecting" similar ones.