Let's say that I'm developing a machine learning system that predicts click-through rate (CTR) for internet ads, that replaces the current heuristic algorithm. The ads with the highest expected revenue (calculated from predicted CTR) is displayed.
Once this algorithm is deployed in production, the properties of learning data are expected to be changed. Previously I was gathering learning data generated from display results made by my heuristic algorithm, that may or may not displays high-revenue ads. Now only the ads with the highest expected revenue are displayed, learning data that we can gather are mostly dominated by these high-revenue ads. As I'm considering periodically update the model, I'd like to use these new learning data, whose properties may be different from the previous learning data.
What kind of problems are expected to happen in this situation? Are there any well-known solution for that problems?