2
$\begingroup$

I'm working with a dataset containing 600 matrices (dim: 44x3) with numerical results of applied tests (pass, fail, toVerify). Data is organized in rows. Row nr 1 represents test nr 1. Tests can be carried out many times on different objects. Columns represent results: P F V. I am going to cluster those matrices without previously specifying number of clusters. I chose agglomerative hierarchical clustering as a point of departure for my research. (Most probably complete-linkage approach) First I normalize dataset by dividing each row element by the total sum of applied tests (sum of row instances). Therefore I obtain data ranging (0:1) so as to be able to compare data. Next I would like to create a dissimilarity matrix and I am wondering which distance metric should be most appropriate in this case. I found a lot of information about measures but there's nearly no speak about their applications. I have 3 proposals and I would be grateful if you could comment on this.

1) use Frobenius norm Would it lose the notion of how data is organized for the sake of 'how big the numbers' in the matrix are?

2) Euclidean norm

3) Vectorization of matrix, applying a Manhattan distance on two vectors a and b and then summing the outcome of each 3 results into one. This approach seems to me most promising for now.

Example (for brevity I analyzed matrices 3x3):

Input matrices: $ A= \left[ \matrix { 5&1&40 \\ 0&2&0 \\ 1&7&2 } \right] $ $ B= \left[ \matrix { 1&1&8 \\ 4&0&0 \\ 2&5&1 } \right] $

Normalization: $ A= \left[ \matrix { 0,1086&0,0217&0,869 \\ 0&1&0 \\ 0,1&0,7&0,2 } \right] $ $ B= \left[ \matrix { 0,1&0,1&0,8 \\ 1&0&0 \\ 0,25&0,625&0,125 } \right] $

At this point I calculated as well Frobenius distance: Fr||A-B|| = $ 1,4299 $

Proceeding with vectorization with regard to rows:

$ a= \left[ \matrix { 0,1086\\ 0,0217\\ 0,869 \\ 0\\ 1\\ 0 \\ 0,1\\ 0,7\\ 0,2 } \right] $ $ b = \left[ \matrix { 0,1 \\ 0,1\\ 0,8 \\ 1\\ 0\\ 0 \\ 0,25\\ 0,625\\ 0,125 } \right] $

Distance $d(a,b)$ is defined as $ L_{1}norm = \sum\limits_{i=1}^9 |a_{i} - b_{i}| $

which results in distance vector d:

$ d = ||a-b|| = \left[ \matrix { 0,0086 \\ 0,0783\\ 0,069 \\ 1\\ 1\\ 0 \\ 0,15\\ 0,075\\ 0,075 } \right] $

In this way I can capture the similarity between rows (each row represents 1 test and. Test can have 3 possible values: passed, verify and failed) as mentioned earlier.

In order to calculate the similarity between 2 instances with focus on rows I add each 3 subsequent elements obtaining finally

$ e = \left[ \matrix { d_{1} + d_{2} + d_{3} \\ d_{4} + d_{5} + d_{6}\\ d_{7} + d_{8} + d_{9} \\ } \right] = \left[ \matrix { 0,1559\\ 2\\ 0,3\\ } \right] $

That's the distance measure between two matrices A and B. This vector is to be placed in a dissimilarity matrix.

I'm quite bewildered which of the metrics would be most applicable in my case.

Any comments on the resolution process would be appreciated since it's my first data mining research (part of the thesis).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.