I am working on a little text clustering problem, and trying to figure out how to evaluate the results. I came up with the following idea that I though fits pretty well with the specifics of the problem. Then I realized that I am probably reinventing the wheel, as surely this is something that has been used in the information retrieval community. If I would present it as an invention of my own, the more knowledgeable crowd would laugh me out of the room. So my question is, what do you call a metric such as the one described below; can you indentify it as an existing metric? (existing in the sense that somebody has given it a name)

Let there be a set of 10 documents, as an example. The system will query for the similarity of one of the documents to all of the other documents. I am not interested in the ranking of all the results to a query of similarity, e.g., given query q, the most similar is result r1, the next is r2, r3, r4, etc. - wouldn't make sense in this context. So what I use as a "gold standard" is a list of two of the (empirically, based on close reading) most similar documents to a preselected set of documents. Close reading the texts takes time, so I do not intend to do this for each of the document; I will take the quality of the rankings of a couple of docs to reflect the overall system quality. I run the program, if the pre-ranked docs rank high on average, then the system is doing well, if they rank low, need to tune system. Note that the concept of relevant/nonrelevant results does not work here, so precision and recall is not what I am looking for (read into that already). All of the documents in the set are relevant in a way, but some are more relevant/similar to one another.

So let's say I run the system and the the following output, which displays the similarity rank of other documents to each of the documents given in the first row:

    d1   d2   d3   d4   d5   d6   d7   d8   d9  d10
1   d8  d10   d9   d9   d3   d2   d5   d1   d3   d6
2   d9   d7   d7   d6   d9  d10   d9   d9   d7   d5
3   d6   d8   d4   d5   d2   d5   d3   d3   d8   d3
4   d4   d6   d8   d8   d6   d4  d10  d10   d5   d9
5   d7   d4   d5   d7   d8   d8   d8   d2  d10   d2
6   d3   d1   d2  d10   d1   d3   d4   d7   d1   d4
7   d2   d3   d6   d1   d4   d7   d1   d5   d2   d1
8   d5   d5  d10   d2  d10   d9   d2   d6   d4   d7
9  d10   d9   d1   d3   d7   d1   d6   d4   d6   d8

Let's say I have composed the following gold standard rankings for documents 1 and 2. Edit: to emphasize, the sets most.similar is not ordered: it makes little sense semantically to say that one member of a given set is more relevant/similar to the target doc than the other.

most.similar(d1) = {d8, d9}
most.similar(d2) = {d3, d6}

The score for d1 is (1+2)/2 = 1.5 (ranks of the these docs). The score for d2 = (7+4)/2 = 5.5. The average score for the system is then (1.5+5.5)/2 = 3.5. Obviously the lower the score, the better. I could also look at the standard deviation of the scores to help me decide.

Note: I first thought about posting it on Stack Overflow, but it seems the question is more about the method/stats than implementation/programming.

  • $\begingroup$ Spearman rank correlatipon? $\endgroup$ – Anony-Mousse Apr 22 '15 at 20:52
  • $\begingroup$ @Anony-Mousse wouldn't that require the compared/correlated vectors to be of the same length? The "standard" I'm comparing against is of length 2 in this example (d8,d9 for d1), while the returned rankings for d1 is of length 9. $\endgroup$ – user3554004 Apr 23 '15 at 11:20
  • $\begingroup$ Use only the intersection. $\endgroup$ – Anony-Mousse Apr 23 '15 at 12:13
  • $\begingroup$ @Anony-Mousse care to elaborate? The intersection of [the set of matches to the d1 in the table] and [set of most.similar(d1)] is just the same {d8,d9}, correlating that is not informative, so I take it you must have somthing else in mind...? $\endgroup$ – user3554004 Apr 23 '15 at 12:23
  • $\begingroup$ Now use only these objects for the correlation. $\endgroup$ – Anony-Mousse Apr 24 '15 at 7:55

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.