I have a application where a logistic regression model is used to make a decision. Each time a user enters the application the model is fed inputs, and it outputs some decision. The first model was trained on data obtained by selecting a decision at random and displaying it to the user. The resulting model performs better than random and has been running online for a while now.
The data of each impression (when a model is given the chance to make a decision) is logged and used for training and comparison of future models
My concern is that future models will be biased because they are being trained on data generated from my current model. Is this concern true? And how can I avoid biasing future models?
One thought: Since the current model is running on 100% of traffic, I could sacrifice maybe 5% to be a random decision. This would allow me to collect unbiased data which I could use to compare models in an unbiased manner. One problem with this approach is that 5% of my data is probably too small a sample to train or test on.
Toy Example with CTR prediction: Lets say app displays a photo to the user. For each opportunity to show a photo, there are several(could be 1000+) photos which the model must select from to show to the user. Each opportunity may have a different set of photos to select from depending on factors. Whether the user clicks on the photo which the model chose is logged and this data is what i plan on using to train models in the future. The LR model is 5-fold cross validated on 70% of the data, and 30% is left for testing.
The problem i am describing here appears to be an inherent problem with models who's prediction is used to make decisions which affect the data that a later iteration of a model would be trained on.
Note: The LR model produces a ranking for all photos which can be shown. then the best ranked is shown.Scores for photos are recomputed every X many days.