2
$\begingroup$

I am trying to cluster my dataset with 15 clusters. As the original labels and the output labels of the K-means algorithm may be different, I am wondering how to find the accuracy. I am using MATLAB kmeans inbuilt library function.

$\endgroup$
  • 3
    $\begingroup$ What you do mean by accuracy? Clustering is an unsupervised learning technique. $\endgroup$ – usεr11852 Jun 18 '17 at 17:48
  • $\begingroup$ The question would be better if you don't refer to matlab. $\endgroup$ – Wayne Sep 19 '17 at 13:16
2
$\begingroup$

Accuracy is not commonly used in unsupervised algorithms.

The problem is that the clustering algorithm does not produce classed, but "1", "2", "3" etc.

The usual evaluation would be to use ARI, Rand (which is similar to accuracy, https://en.wikipedia.org/wiki/Rand_index), NMI, etc.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 I hadn't heard of the Rand Index -- and ARI is "Adjusted Rand Index", it turns out -- but I've "invented" it before and it's the answer to the question. I added a link to Wikipedia for you. You might want to expand a bit on what Rand does, so the answer is a bit more complete for future reference. $\endgroup$ – Wayne Sep 19 '17 at 13:14
  • $\begingroup$ I think people should rather look up Rand/ARI in literature than rely on an attempt of mine to summarize it again here... $\endgroup$ – Has QUIT--Anony-Mousse Sep 19 '17 at 18:59
  • 1
    $\begingroup$ True, BUT one of the principles of Stack Exchange is that short answers that point you elsewhere are not the best answers. Links age off. I googled "ARI" along with other statistical words and got nothing. Only when I finally found a description of the Rand Index did I magically find that ARI is a variant of it. And so on. Unexpanded 3-letter acronyms are vary hard to figure out or search, Please fix this so I can maintain my +1 for your answer. $\endgroup$ – Wayne Sep 19 '17 at 20:32
  • $\begingroup$ Google "clustering ari" works fine, "kmeans ari" too. and any clustering book should have this in the index, too. Yes, links age, I didn't put any (but so does code, and stackoverflow has tons of code that no longer works with current versions). $\endgroup$ – Has QUIT--Anony-Mousse Sep 19 '17 at 23:09
-1
$\begingroup$

When the machine learning problem is prediction, then there can be an automated accuracy/precision measurement. However, in many other problems, you need to compare the machine's choice $m$ against human choice $h$.

A basic accuracy score on a test set of $N$ elements is naturally percentage of match :

$$score = \frac{1}{N}\sum_{i=1}^N 1_{m(i)=h(i)}$$

Measuring accuracy of non-supervized learning is a bit tricky and there is no definition or solution out of the box: the machine invents its own categories, you invent yours. They may not be the same, it does not mean the machine's choice is not as meaningful as yours.

My team did a certain work : categorization of texts into "topics". First we created a small dataset with 5 very different topics chosen by us. We said the algo was perfectly accurate if it divided the texts into the same topics as we did, and agreed for each text. Actually, since the topics were clearly different, accuracy was close to 98%

But in real life datasets, when you don't have clearly non-overlaping categories, things are not so simple. Deciding to divide according to politics/economics/people may be as "good" as dividing into employment/elections/cinema for example.

We found a way to overcome this problem with a sort of semi-supervision. But it's not a universal solution. We decided of a predefined set of 24 categories : "art&entertainment", "sports"... and ran a k-means with 100 clusters : more than the categories. We had actually many texts : > 100 000 so that 100 clusters was still a small number. Then we decided to relate each cluster to categories (thanks to "important" words in the cluster for example) manually:

  • cluster 1 (car, race, schumacher...) : "automotive"
  • cluster 2 (audi, bmv, nissan...) : "automotive"
  • cluster 3 (funk, jazz, concert...) : "art&entertainment"
  • cluster 4 (dali, picasso, louvres...) : "art&entertainment"
  • ...

So the predicted category was obtained in two steps :

text ---(automated)---> cluster ----(manual)---> category

Then we took a test set of say 1000 texts and chose the category of each text by reading the text. We compared the predicted category (closest center -> category) versus our own human choice and calculated a score : match percentage.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.