At the moment I'm researching clustering of single words. The input of this research is a list of words (unigrams). During the research I want to compare several similarity algorithms to see how they impact the cluster result.
At the moment I’m using a 70word data set, where the words belong to 7 clusters (i.e., animals, music instruments, vehicles etc). How do I evaluate the different algorithms? I can use measures like: Rand measure, F-measure or Jaccard index. But I’m struggling with two questions. The first one is the cluster count and the second how do you measure accuracy?
In order to compare you need the same measure, otherwise what you are comparing to each other doesn’t say anything. Thus, I need to keep the cluster count equal, or can I compare while using different cluster counts? The silhouette function in Matlab shows me that the optimal cluster count differ between the similarity algorithms, when I look at the data I see a lot of errors and this error count will go down when I choose more clusters (this is no surprise).
Second is the accuracy measure. How do I determine if the word is in the right cluster? Because it is more a data exploration tool. For example with the data set from above and I choose to cluster the words into seven clusters I got the following result: six vehicles and one animal. Is my precision and recall then .86? Or do I need to calculate this over all clusters? If so, how do I cope with clusters that have six vehicles and six animals? What is the right cluster in order to determine if the right word is inside? Or doesn’t it matter because the math will stay the same?
