Remove bias in ranking evaluation In ranking, a commonly used method to evaluate the performance of a ranking algorithm is calculating ranking metrics such as MAP, NDCG. In use cases where there is no ground truth signal, some proxy signals as used for relevance. For example, in a recommendation system, we can use user's interactions (clicks, purchases) with the items as the positive relevance signal to calculate MAP or NDCG.
The problem with this is that the interactions(clicks/purchases in the example) has bias toward the ranker (usually the production ranker) that generates them. Thus it's not fair to compare the production ranker and a new ranker using this kind of signal. Is there a way to de-bias such signals for evaluation so that one can fairly compare new algorithms with the one that produce the evaluation data?
 A: This is a great question that people are doing research on. Therefore, part of my answer will be throwing papers at you.


*

*Offline vs. Online. (The get more labels approach)


I assume that you want to correct these labels so you can do an offline evaluation. One of the strategies for dealing with this issue is an "interleaved" evaluation where you present a fusion of results between two systems in order to compare them -- basically it boils down to which system gets more clicks at that point.
Some of this comes down to the "Exploration vs. Exploitation" tradeoff. Could you use your algorithm pool to make sure that users see candidates that will be more informative? (Hofmann, Katja, Shimon Whiteson, and Maarten de Rijke. "Balancing Exploration and Exploitation in Learning to Rank Online.")


*Weighting instances by how likely they are to be rank-biased. (Learn better model anyway / re-weight by surprise)


It turns out you can learn weights on queries (clicks) based on how far down they are. Basically, the more they diverge from your production ranker, the more informative they are for learning (under the hypothesis that your prod ranker will get easy queries right). There are sophisticated strategies for this, e.g., (Wang, Xuanhui, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. "Position Bias Estimation for Unbiased Learning to Rank in Personal Search." (2018).) but they cite a paper that used simple weights learned from randomization experiments.


*The evaluation-metric way (focus on actual clicks somehow)


Throw out all documents from your rankings that don't have clicks (or views), and calculate MAP only on those. This is fair, but difficult, potentially, with only positive labels (you could infer some negative weight on documents that appeared before a click but were not clicked themselves). There are a number of ways to do this, from the heuristic, to special evaluation signals:
Sakai, Tetsuya. "Alternatives to bpref." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2007.
+ Google Scholar results for "unjudged documents sigir"
Conclusion
The most robust answer will involve an online experiment. We know this from statistics. You're never supposed to use a dataset twice. 
However, depending on your position (industry v. academia) it may or may not be possible to simply do another experiment, or to put an untrusted ranker in front of users to get more fair labels. In which case, try to zoom in only on the biased labels (Option 3) or weight them using (Option 2) and use NDCG. These approaches can be justified and are slightly better than reinventing the wheel.
