# Transforming confusion matrix to distance matrix

I have an annotation task which I need to analyze for accuracy, category fine tuning and reliability , there are many annotators assigning multiple set valued categories to items. And I have come up with confusion matrices for each pair of annotators like below:

+-----------------------------------------+
|  |5|  .   1   .   .   .   .   2   .   . |
|   7  |.|  .   .   .   .   .   5   .   1 |
|  34   .  |.|  .   .   .   1   4   .   1 |
|   7   1   .  |.|  .   .   1  46   .   . |
|   .   .   .   .  |.|  1   .   .   .   . |
|   5   .   .   .   .  |.|  .   1   .   . |
|   5   .   .   .   .   .  |.|  .   .   . |
|   5   .   .   .   .   .   1  |2|  .   5 |
| 198   .   1   .   .   2   1  11  |.|  1 |
|   1   .   .   .   .   .   2   .   .  |.||
------------+-----------------------------+


To transform it into an undirected distance matrix , I transposed the above and averaged it to get the following symmetric matrix which I need to represent as distance matrix:

|   |5|   3   17    3    .    2    2    3   99    0 |
|    3   |.|   .    0    .    .    .    2    .    0 |
|   17    .   |.|   .    .    .    0    2    0    0 |
|    3    0    .   |.|   .    .    0   23    .    . |
|    .    .    .    .   |.|   0    .    .    .    . |
|    2    .    .    .    0   |.|   .    0    1    . |
|    2    .    0    0    .    .   |.|   0    0    1 |
|    3    2    2   23    .    0    0   |2|   5    2 |
|   99    .    0    .    .    1    0    5   |.|   0 |
|    0    0    0    .    .    .    1    2    0   |.||


I have about 190 such matrices from multiple annotator pairs and there is nothing like reference and test categories as the annotator category assignment needs to be analyzed for fine tuning, merging the categories.

So is the above approach correct to start with and how to transform multiple such matrices (dimensions)into a single visualization allowing to analyze the categories and inter annotator reliability apart from PCA and the usual kappa and related statistics?

Also would such a distance matrix give a correct indication of the real components in PCA for cluster analysis to base further decisions on?