How do you update a model when the implementation of your model eliminates new data?
I have created a boosted trees classification model that predicts whether or not the amount of money requested (
$X) in a given funding application would be different from the amount actually approved (
$Y) after a review process (does
$X == $Y). The goal is to implement an automatic review system to immediately clear funding applications which are not expected to see a change after human review.
The model performs excellently right now. I am wondering how to keep it updated once it is put into practice. For example, total amount of funding requested is highly impactful on the prediction, but I expect funding requests to increase with inflation or change based on business rules. Or perhaps in the future entirely new features may need to be incorporated. But after this model is used there won't be any new data to train on--the only applications that will undergo human review are those applications that are predicted to have
$X != $Y. Applications marked "auto-approve" will automatically have
$Y set to equal
$X. I will have no unbiased source of new data.
What to do? Should I ask that a certain percentage of applications be randomly sampled to undergo human review (whether or not they're marked "auto-approved") in order to be used as training data for the next year? How would I determine that percentage? Is there another way?