Avoiding Bias Through Model Selection

Question

I have a application where a logistic regression model is used to make a decision. Each time a user enters the application the model is fed inputs, and it outputs some decision. The first model was trained on data obtained by selecting a decision at random and displaying it to the user. The resulting model performs better than random and has been running online for a while now.

The data of each impression (when a model is given the chance to make a decision) is logged and used for training and comparison of future models

My concern is that future models will be biased because they are being trained on data generated from my current model. Is this concern true? And how can I avoid biasing future models?

One thought: Since the current model is running on 100% of traffic, I could sacrifice maybe 5% to be a random decision. This would allow me to collect unbiased data which I could use to compare models in an unbiased manner. One problem with this approach is that 5% of my data is probably too small a sample to train or test on.

Toy Example with CTR prediction: Lets say app displays a photo to the user. For each opportunity to show a photo, there are several(could be 1000+) photos which the model must select from to show to the user. Each opportunity may have a different set of photos to select from depending on factors. Whether the user clicks on the photo which the model chose is logged and this data is what i plan on using to train models in the future. The LR model is 5-fold cross validated on 70% of the data, and 30% is left for testing.

The problem i am describing here appears to be an inherent problem with models who's prediction is used to make decisions which affect the data that a later iteration of a model would be trained on.

Note: The LR model produces a ranking for all photos which can be shown. then the best ranked is shown.Scores for photos are recomputed every X many days.

Answer 1

The answer regarding bias is YES, using your model for in-the-loop decision support influences the outcome. Data modeling almost invariably starts from the treatment of observations as a dependent variable, with independent variables that interact in an unknown system to produce the observed results. Modeling approaches attempt to infer or estimate from observation the hidden workings of the generating system (with wide variations in approaches and assumptions). In this case, your existing LR model is part of the generating system (since it leads to decisions that influence outcomes), so any attempt to model this system by other means will reflect its ability to predict, in part, how your existing LR model impacts observed outcomes.

Your application is almost more in the domain of control theory rather than pure data modeling, since you are using a combination of a predictive model and a control signal to shape outcomes. "Pure" data modeling attempts to predict outcomes from input values, while your application attempts to dictate them based on probabilistic predictors.

This is the case for many real-world problems though. In such cases, the real objective is not to get the best predictor of outcomes, but rather to optimize the expected outcomes of decisions that are conditioned upon a predictive model of the response to specific inputs (which now include decision variables that are in the system's control). This plays into stochastic optimization and decision theory, which live outside the domain of strict data modeling.

Answer 2

Bias is perhaps not the right term. The model estimated effects are "true" in the sense that they are internally valid, or that they "generalize" only to those data which you've collected (not any kind of generalization at all). The more pertinent issue is that they may not generalize to external data which would be collected in a follow up or independent cluster. Those independent data would give rise to a different model, but the difference between those models would not be bias, but just difference between two internally valid models with no external validity. A fitting method which estimates the same effects in both independent datasets could be said to be externally valid. Otherwise the method is flawed.

The truth is your model is overfitted. This leads to biased effect estimates, but the bias is not the issue. The fact is that this model is not guaranteed to generalize to future or separate data collection, it is not externally valid.

This is a bigger issue than just cross validation, its an issue of defining the scope of your model. Nonetheless, cross validation will help relieve the bias, partially by encouraging you to fit a model on a smaller fraction of the total training set. A better approach yet is to focus on the science of the problem and what is known about the larger temporal and spatial frame whence these data came.