4
$\begingroup$

Two sets of comparisons were performed between original clustering and the new clustering using several indices and metrics of performance. Below are the two initial clusterings or partitions (these should be the truth or original partitions), they are two because samples were taken from two different locations and that's why we have always two sets of partitions:

enter image description here

The image below shows the new clusterings obtained by utilizing only a subset of data for each location instead of the whole dataset and then were compared with the original ones (obtained using the whole dataset), i.e., left original with left new, right original with right new.
enter image description here

The two partitions were compared with their original coutner part (left with left, right with right) as I said and below were the results: 

        Name    Part1 (Lt) Part2 (Rt)
1        Purity 0.9633028  0.7431193
2            VI 0.2451685  1.1486369
3           NMI 0.8673525  0.4062956
4    Split/Join 8.0000000 62.0000000
5 Adjusted Rand 0.8750403  0.2131243
6          Rand 0.9374788  0.6083928
7       Jaccard 0.8800522  0.4045466

Note:
VI short for "Variation of Informaiton"
NMI short for "Normalized Mutual Information"
ARI short for "Adjusted Rand Index"
split/join is also known as van Dongen S

So it clear that the left new partition is much similar to its left original counterpart partition. The right new partition was way different from its right original counterpart. Samples were numbered from 1 to 34 (y-axis) and were sampled at five different occasions (x-axis). Purity measure was in line with the above and is easy to grasp. The other indices/metrics as far as I know, each measures a different thing. But to be honest as a non-statistician I cannot relate to the extensive math behind each of them. So my request from the community is how to interpret these indices with respect to the partitions in the images below without using fancy statistical terms (I have already came across a bunch of them elsewhere) but rather in simple plain English.

For example (pls correct me if I am wrong), I learned the following:

  1. Jaccard distance measures how far one partition from the other, it can be obtained by 1-Jaccard index, which makes sense
  2. for VI; it measures rather difference than similarity, its values is not between 0 and 1 like NMI, it is something between 0 and not more than $2log k$. $k$ is the number of clusters so if 4 as in this case the value should not be more than 2.log4 = 2.772589.
  3. NMI is between 0 and 1 (NMI = 0, two partitions contain no information about one another, NMI = 1, two partitions contain perfect information about one another)
  4. ARI is the same as Rand Index but corrected for chance. This measure is zero when the Rand index takes its expected value, and has a maximum of one.

Question:
I would be very grateful to have an intuitive interpretation for Split/Join metric (as proposed by Micans, also known as van Dongen S metric), VI what does it measure exactly in relation to the images of the example shown, Jaccard how different is it from Rand? Pls feel free to correct any misconceptions posted in this question. I want to grasp the intuition behind and learn from you the correct way to interpret these measures and how they could help understand partitions from their different angles of view. Having results is fine but commenting on them is an art requires deep understanding of the tools and the philosophy behind them, may I find this with you? I hope.

Improve this question
$\endgroup$
1
  • $\begingroup$ Link to more information about the split/join metric by Stijn van Dongen. $\endgroup$
    – rlchqrd
    Mar 24, 2021 at 15:52

1 Answer 1

Reset to default
4
$\begingroup$

The split/join metric measures the number of 'moves' required to go from the first clustering to the second clustering, where each 'move' consists of splitting off a single element off of one cluster and then either attaching it to another cluster (which also counts as a move) or starting a new cluster. There is a further requirement, which is not very important for the intuition, that these moves are 'aligned' with the lattice of partitions. This means that the path sketched out by the 'moves' also contains the largest common subclustering of the two clusterings. In your case, a single node or element is, I assume, a single cell. The intuition thus is, that for the left instance, eight cells need to be rearranged in order to obtain one of the clusterings from the other. The question below and its answers may also be interesting: Comparing clusterings: Rand Index vs Variation of Information

Improve this answer
$\endgroup$
5
  • $\begingroup$ thanks quite interesting what you just said about split/join. I fell in love with this metric. Pls stay with the Lt side, the metric was 8, you said they denote 8 "moves" of cells, but visually I spotted 4 mismatches, as (x,y): (4,5),(2,10),(4,23),(2,32). So if 8 this means two-way direction of movements, to and from, the two clusters to get two fully matching clusters from each other regardless which cluster was parent or child, interesting! can you pls confirm? other issue, could you pls comment on my conceptions about the metrics mentioned in 4 points if true. I would be grateful. $\endgroup$
    – doctorate
    Dec 19, 2013 at 17:56
  • $\begingroup$ can we say that apart from the split/join and VI metrics, all values of the above mentioned indices/metrics would lie between 0 and 1? $\endgroup$
    – doctorate
    Dec 19, 2013 at 18:10
  • $\begingroup$ Regarding the two-way direction, you are correct. I advocate to write out the split/join distance in two parts (this is possible in igraph) as a+b, where a and b denote the respective distances to the largest common subclustering. This will identify cases where one clustering is a subclustering of the other or nearly so. Other distances such as VI and 'mirkin' (which is the complement of rand) have the same trait. Your four points are all correct. I find ARI a bit mystifying, and a strangely and unnecessarily probabilistic take on what is a beautiful metric space (the space of partitions). $\endgroup$
    – micans
    Dec 20, 2013 at 11:59
  • $\begingroup$ Note that VI and split/join can also easily be scaled to lie inbetween 0 and 1 (by using appropriate scaling factors), just like the others. This would make it a bit easier to compare different criteria. For VI the natural factor is 2 log(k), for split/join it is 2 N where N is the number of elements (cells in your case). The thus normalised split/join distance can of course be zero (when clusterings are identical). It can never be exactly 1, but for large N it can get very close to 1 (for maximally disagreeing pairs of clusterings). $\endgroup$
    – micans
    Dec 20, 2013 at 12:18
  • 1
    $\begingroup$ thanks for your insightful comments, if $N$ is 109, then normalized split/join as you mentioned above would be 8/218 = 0.037, and 68/218= 0.31, resp. Not too much intuitive to be gained, I guess. I think the beauty of split/join shines more when unscaled. $\endgroup$
    – doctorate
    Dec 20, 2013 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.