The idea behind Pairwise Learning to Rank is that if you have a set of search results then a clicked on result can be used as training example to indicate that it should rank more highly that the results above it which were not clicked on.

Let's say you have a search engine, which you collect a lot of click data for. Now you tweak the algorithm and run it against your training set to see if it's an improvement to the algorithm. We know which examples it does better on because it places some clicked items above non-clicked items. How do we know where it is doing worse?

  • $\begingroup$ Maybe far to trivial, but how about inversions? In this reference it is actually positioned the number to minimise: en.wikipedia.org/wiki/Learning_to_rank#Pairwise_approach $\endgroup$ – spdrnl May 15 '15 at 12:55
  • $\begingroup$ @spdrnl the issue is that it only lets you know when you're doing better - not worse. If a clicked-on item moves above a non-clicked item, then you're doing better. But what if the algorithm is worse in nearly all cases, but in a few cases moves a clicked on item above a non-clicked on item? How do you know that this second algorithm is worse? $\endgroup$ – Ben McCann May 15 '15 at 18:44
  • $\begingroup$ I think I grasp it after some (re-)consideration. If the new algorithm is better, then you expect users to click on items that are nearer to the top. Counting from the top, the average position should be nearer to 1. Using an A/B test for example one should be able to find out which algorithm is doing better and which algorithm is doing worse. (This implies I think that there are more observed inversions.) $\endgroup$ – spdrnl May 15 '15 at 18:58
  • $\begingroup$ Good question.My boss had ask same question. In my suggestion, it depends on what business you do. My job can be describe as rank goods in some website like Amazon in China. When we do ABtest for a new model, not only concern about cvr, ctr. Also, the click goods uv, the buyer uv, Matthew of goods all are important. $\endgroup$ – whb_zju May 10 '16 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.