Disadvantage of precision at k

Question

Suppose 10 documents were retrieved (rectangle with black color is relevant document). In the following table, Precision @ k is calculated. P@10 or "Precision at 10" corresponds to the number of relevant results among the top 10 documents.

| rank(K)| Precision@K    |
| ------ | -------------  | 
| 1      | (1/1) = 1.0    |   
| 2      | (1/2) = 0.5    |  
| 3      | (2/3) = 0.67   | 
| 4      | (3/4) = 0.75   | 
| 5      | (4/5) = 0.80   | 
| 6      | (5/6) = 0.83   | 
| 7      | (6/7) = 0.86   | 
| 8      | (6/8) = 0.75   | 
| 9      | (7/9) = 0.78   | 
| 10     | (7/10) = 0.70  | 

The more relevant documents are at the front, the more influence it has and earn more points. So then we have greater Precision@k. So, if we swap ranks 8 and 9 we get the following results:

| rank(K)| Precision@K    |
| ------ | -------------  | 
| 1      | (1/1) = 1.0    |   
| 2      | (1/2) = 0.5    |  
| 3      | (2/3) = 0.67   | 
| 4      | (3/4) = 0.75   | 
| 5      | (4/5) = 0.80   | 
| 6      | (5/6) = 0.83   | 
| 7      | (6/7) = 0.86   | 
| 8      | (7/8) = 0.875  | 
| 9      | (7/9) = 0.78   | 
| 10     | (7/10) = 0.70  | 

Wikipedia says: Precision at k documents (P@k) is still a useful metric (e.g., P@10 or "Precision at 10" corresponds to the number of relevant results among the top 10 documents), but fails to take into account the positions of the relevant documents among the top k.

Unfortunately I don't get the last sentence. How does it mean here? We could even see that changing the position changes the precision @ k.

Answer 1

You need to understand

but fails to take into account the positions of the relevant documents among the top k

in the context of a fixed $k$. In your example, with $k=10$, precision at $k=10$ does not change if we swap ranks 8 and 9, as we see in the bottom rows in your tables.

Conversely, precision at $k=8$ does change, because here it is not just a question of the relative position among the top $k=8$ - rather, the document either is among the top $k=8$, or it isn't. (And among the top $k=8$, order again does not matter.)

Answer 2

That's a great example by Stephan. To expand a bit more, the idea with Precision @ K is that you measure the presence or absence of true results within the top K results, independent of where they are.

That last sentence

• but fails to take into account the positions of the relevant documents among the top k

means that the rank or the order of the observations is not taken into account by a metric like Precision @ k. Meaning, if you care about the order or position of where the relevant documents are (1st,3rd, 8th, etc..) then, this metric isn't useful.

Specifically, here's an example. Suppose, you have 10 true relevant documents among the top 20 hits, it won't tell you where those 10 are. They could be the top 10, or they could be documents 11-20. Precision at K=20, will only tell you you have 50% precision at K=20. However, if you test a few different Ks you would figure out that now at K=18, you'll have precision=8/18, and at K=10, you'll have 0 precision. But if you only test K=20, you wouldn't know where they are.