# An IR evalualuation metric that only measures the rank of results?

I am working on a little text clustering problem, and trying to figure out how to evaluate the results. I came up with the following idea that I though fits pretty well with the specifics of the problem. Then I realized that I am probably reinventing the wheel, as surely this is something that has been used in the information retrieval community. If I would present it as an invention of my own, the more knowledgeable crowd would laugh me out of the room. So my question is, what do you call a metric such as the one described below; can you indentify it as an existing metric? (existing in the sense that somebody has given it a name)

Let there be a set of 10 documents, as an example. The system will query for the similarity of one of the documents to all of the other documents. I am not interested in the ranking of all the results to a query of similarity, e.g., given query q, the most similar is result r1, the next is r2, r3, r4, etc. - wouldn't make sense in this context. So what I use as a "gold standard" is a list of two of the (empirically, based on close reading) most similar documents to a preselected set of documents. Close reading the texts takes time, so I do not intend to do this for each of the document; I will take the quality of the rankings of a couple of docs to reflect the overall system quality. I run the program, if the pre-ranked docs rank high on average, then the system is doing well, if they rank low, need to tune system. Note that the concept of relevant/nonrelevant results does not work here, so precision and recall is not what I am looking for (read into that already). All of the documents in the set are relevant in a way, but some are more relevant/similar to one another.

So let's say I run the system and the the following output, which displays the similarity rank of other documents to each of the documents given in the first row:

 doc
d1   d2   d3   d4   d5   d6   d7   d8   d9  d10
1   d8  d10   d9   d9   d3   d2   d5   d1   d3   d6
2   d9   d7   d7   d6   d9  d10   d9   d9   d7   d5
3   d6   d8   d4   d5   d2   d5   d3   d3   d8   d3
4   d4   d6   d8   d8   d6   d4  d10  d10   d5   d9
5   d7   d4   d5   d7   d8   d8   d8   d2  d10   d2
6   d3   d1   d2  d10   d1   d3   d4   d7   d1   d4
7   d2   d3   d6   d1   d4   d7   d1   d5   d2   d1
8   d5   d5  d10   d2  d10   d9   d2   d6   d4   d7
9  d10   d9   d1   d3   d7   d1   d6   d4   d6   d8


Let's say I have composed the following gold standard rankings for documents 1 and 2. Edit: to emphasize, the sets most.similar is not ordered: it makes little sense semantically to say that one member of a given set is more relevant/similar to the target doc than the other.

most.similar(d1) = {d8, d9}
most.similar(d2) = {d3, d6}


The score for d1 is (1+2)/2 = 1.5 (ranks of the these docs). The score for d2 = (7+4)/2 = 5.5. The average score for the system is then (1.5+5.5)/2 = 3.5. Obviously the lower the score, the better. I could also look at the standard deviation of the scores to help me decide.

Note: I first thought about posting it on Stack Overflow, but it seems the question is more about the method/stats than implementation/programming.

• Spearman rank correlatipon? – Anony-Mousse Apr 22 '15 at 20:52
• @Anony-Mousse wouldn't that require the compared/correlated vectors to be of the same length? The "standard" I'm comparing against is of length 2 in this example (d8,d9 for d1), while the returned rankings for d1 is of length 9. – user3554004 Apr 23 '15 at 11:20
• Use only the intersection. – Anony-Mousse Apr 23 '15 at 12:13
• @Anony-Mousse care to elaborate? The intersection of [the set of matches to the d1 in the table] and [set of most.similar(d1)] is just the same {d8,d9}, correlating that is not informative, so I take it you must have somthing else in mind...? – user3554004 Apr 23 '15 at 12:23
• Now use only these objects for the correlation. – Anony-Mousse Apr 24 '15 at 7:55