This is a great question that people are doing research on. Therefore, part of my answer will be throwing papers at you.
- Offline vs. Online. (The get more labels approach)
I assume that you want to correct these labels so you can do an offline evaluation. One of the strategies for dealing with this issue is an "interleaved" evaluation where you present a fusion of results between two systems in order to compare them -- basically it boils down to which system gets more clicks at that point.
Some of this comes down to the "Exploration vs. Exploitation" tradeoff. Could you use your algorithm pool to make sure that users see candidates that will be more informative? (Hofmann, Katja, Shimon Whiteson, and Maarten de Rijke. "Balancing Exploration and Exploitation in Learning to Rank Online.")
- Weighting instances by how likely they are to be rank-biased. (Learn better model anyway / re-weight by surprise)
It turns out you can learn weights on queries (clicks) based on how far down they are. Basically, the more they diverge from your production ranker, the more informative they are for learning (under the hypothesis that your prod ranker will get easy queries right). There are sophisticated strategies for this, e.g., (Wang, Xuanhui, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. "Position Bias Estimation for Unbiased Learning to Rank in Personal Search." (2018).) but they cite a paper that used simple weights learned from randomization experiments.
- The evaluation-metric way (focus on actual clicks somehow)
Throw out all documents from your rankings that don't have clicks (or views), and calculate MAP only on those. This is fair, but difficult, potentially, with only positive labels (you could infer some negative weight on documents that appeared before a click but were not clicked themselves). There are a number of ways to do this, from the heuristic, to special evaluation signals:
Sakai, Tetsuya. "Alternatives to bpref." Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2007.
+ Google Scholar results for "unjudged documents sigir"
The most robust answer will involve an online experiment. We know this from statistics. You're never supposed to use a dataset twice.
However, depending on your position (industry v. academia) it may or may not be possible to simply do another experiment, or to put an untrusted ranker in front of users to get more fair labels. In which case, try to zoom in only on the biased labels (Option 3) or weight them using (Option 2) and use NDCG. These approaches can be justified and are slightly better than reinventing the wheel.