When using search engine query logs to make use of click-through data to order search results, there is a well-known position bias. This position bias gives search engine users a tendency to click on the top most links in the search engine results page, since they scan the results from top to bottom.
I want to know if there has been any work where researchers have presented the Top-N search engine results in a stochastic manner, in order to elicit the most popular search results by looking at the ensuing query logs for click-through rates.
To be more specific about the stochastic nature of presentation of the results, what I mean is that the researcher takes the top 50 results (for example) and mixes them up so that they each have an equal chance of appearing at the top of the results page. To my mind, this should remove the position bias and give a cleaner signal of which of the Top 50 results the web users truly find useful.