I am currently looking for a scientific reference (journal article, book etc) that describes the challenge of evaluating the performance of a method on the data it produces itself.
A trained ranking algorithm A in a production system is producing ranked result lists of documents for a set of users. These users are asked to use documents of these results lists in their work.
Come in algorithm B that is supposed to be evaluated against algorithm A. As an evaluation, it is decided to compare both algorithms by their ranking of the user-chosen documents in a defined time frame.
Although these data points were not part of the training, the results will highly likely be in favor for algorithm A, as it was used to produce the result list the user then selected the documents from.
The closes description I could come up with is "self-fulfilling prophecy bias", but I was unsuccessful in finding anything around this yet.