I am developing a random forest for a binary classification problem where the trained data is heavily skewed towards one class (90% is class A and 10% is class B). The model scores data points based on their likelihood to be of class A, i.e., the number of trees in the forest that classify the data point as class A. This scoring effectively ranks data from the point that is most likely to be class A (top of the list), to the least likely (bottom of the list).
The training data is generated by another system on a monthly basis, and human operators review the data labeling each point as A or B. Suppose this model is used in a business process in such a way that the top 20% of the scored list (most likely to be A) is automatically labeled as A, and the remaining 80% is manually labeled as usual. This is going to introduce some errors in the automatically generated labels, and possibly more and more if we consider for automatic labeling increasing portions of the list (30% and above). However, if the model is good enough, we can live with such errors that define acceptable thresholds of precision, recall, accuracy, or whatever performance metric.
Since the manual labeling process typically changes over time, the model needs to be retrained every now and then. However, the training data will progressively contain less and less "obvious" examples of class A, because they will be excluded from the manual labeling process – effectively introducing a selection bias in the training data. This is of course unless we want to also learn from the previous model outputs (the 20% of data points labeled automatically), which intuitively doesn't sound very smart.
How do you expect the model to perform with such a selection bias in training data? What could happen if, rather than excluding from manual labeling 20% of data, we exclude 70% or 80%? The question might sound a little too generic, but this seems to me an interesting theoretical problem closely related to the application of Machine Learning approaches to real-world problems. I have never stumbled upon any paper about this. Maybe it also probably has a specific name, better than the generic "selection bias in training data", but I don't recall seeing this in the past.