How to handle a "self defeating" prediction model? I was watching a presentation by an ML specialist from a major retailer, where they had developed a model to predict out of stock events. 
Let's assume for a moment that over time, their model becomes very accurate, wouldn't that somehow be "self-defeating"? That is, if the model truly works well, then they will be able to anticipate out of stock events and avoid them, eventually getting to a point where they have little or no out of stock events at all. But then if that is the case, there won't be enough historical data to run their model on, or their model gets derailed, because the same causal factors that used to indicate a stock out event no longer do so. 
What are the strategies for dealing with such a scenario? 
Additionally, one could envision the opposite situation: For example a recommender system might become a "self-fulfilling prophecy" with an increase in sales of item pairs driven by the output of the recommender system, even if the two items aren't really that related. 
It seems to me that both are results of a sort of feedback loop that occurs between the output of the predictor and the actions that are taken based on it. How can one deal with situations like this?
 A: Presumably you can track when restock events happen. Then it's just a matter of arithmetic to work out when the stock would be depleted had the model not been used to restock inventory.
This assumes that any positive stock level is independent of the level of sales. A commenter says that this assumption doesn't hold in reality. I don't know either way -- I don't work on retail data sets. But as a simplification, my proposed approach permits one to make inferences using counterfactual reasoning; whether or not this simplification is too unrealistic to give meaningful insight is up to you.
A: Your scenario bears a lot of resemblance to the Lucas Critique in economics.  In machine learning, this is called "dataset shift".  
You can overcome it, as @Sycorax says, by explicitly modeling it.
A: If you are using a model to support decisions about intervening in a system, then logically, the model should seek to predict the outcome conditioned on a given intervention. Then separately, you should optimize to choose the intervention with the best expected outcome. You are not trying to predict your own intervention.
In this case, the model could predict demand (the variable you don't directly control) and this, in combination with the stocking choice, would result in having an out-of-stock event or not. The model should continue to be "rewarded" for predicting demand correctly since this is its job. Out-of-stock events will depend on this variable along with your stocking choice.
A: One thing to remember is that ML is an instrumental goal. Ultimately, we don't want to predict out of stock events, we want to prevent out of stock events. Predicting out of stock events is simply a means to that end. So as far as Type II errors are concerned, this isn't an issue. Either we continue to have OOSE, in which case we have data to train our model, or we don't, in which the problem that the model was created to address has been solved. What can be a problem is Type I errors. It's easy to fall into a Bear Patrol fallacy, where you have a system X that is built to prevent Y, you don't see Y, so you conclude that X prevents Y, and any attempts to shut X down are dismissed on the basis "But it's doing such a good job preventing Y!" Organizations can be locked into expensive programs because no one wants to risk that Y will come back, and it's difficult to find out whether X is really necessary without allowing that possibility. 
It then becomes a trade-off of how much you're willing to occasionally engage in (according to your model) suboptimal behavior to get a control group. That's part of any active exploration: if you have a drug that you think is effective, you have to have a control group that isn't getting the drug to confirm that it is in fact effective.
A: There are two possibilities by which an out-of-stock (OOS) detection model might self-derail:

*

*The relationship between inputs and OOS might change over time. For instance, promotions might lead to higher OOS (promotional sales are harder to predict than regular sales, in part because not only average sales increase, but also the variance of sales, and "harder-to-predict" translates often into OOS), but the system and its users might learn this and lay in additional stock for promotions. After a while, the original relationship between promotions and OOS does not hold any more.
This is often called "model shift" or similar. You can overcome it by adapting your model. The most common way is to weight inputs differently, giving lower weight to older observations.


*Even if the relationship between a predictor and OOS does not change, the predictor's distribution might. For instance, multiple days with zero sales of a particular stock keeping unit (SKU) might signal an OOS - but if the model performs well, then OOS might be reduced across the board, and there might simply not be as many sequences of zero sales.
Changes in the distribution of a predictor should not be a problem. Your model will simply output a lower probability of OOS.

In the end, you probably don't need to worry overmuch. There will never be zero OOS. Feedback mechanisms like the ones above do occur, but they will not work until OOS are completely eradicated.

*

*Some pending OOS may simply not be avertable. "I have one unit on the shelf and will probably face a demand for five over the coming week, but the next delivery is only due a week from today."

*Some OOS will be very hard to predict, even if they are avertable, if they had been known in time. "If we had known we would drop the pallet off the forklift and destroy all the product, we would have ordered another one."

*Retailers do understand that they need to aim for a high service level, but that 100% are not achievable. People do come in and buy up your entire stock on certain products. This is hard to forecast (see above) and sufficiently rare that you do not want to fill up your shelves on the off chance this might happen. Compare Pareto's law: a service level of 80% (or even 90%) is pretty easy to achieve, but 99.9% is much harder. Some OOS are consciously allowed.

*Something similar to Moore's law holds: the better ML becomes, the more expectations will increase, and the harder people will make life for the model. While OOS detection (and forecasting) algorithms improve, retailers are busy making our life more difficult.

*

*For instance through variant proliferation. It's easier to detect OOS on four flavors of yoghurt than on twenty different flavors. Why? Because people don't eat five times as much yoghurt. Instead, pretty much unchanged total demand is now distributed across five times as many SKUs, and each SKU's stock is one fifth as high as before. The Long Tail is expanding, and signals are getting weaker.

*Or by allowing mobile checkout using your own device. This may well lower psychological barriers to shoplifting, so system inventories will be even worse than they already are, and of course, system inventories are probably the best predictor for OOS, so if they are off, the model will deteriorate.



I happen to have been working in forecasting retail sales for over twelve years now, so I do have a bit of an idea about developments like this.

I may be pessimistic, but I think very similar effects are at work for other ML use cases than OOS detection. Or maybe this is not pessimism: it means that problems will likely never be "solved", so there will still be work for us even decades from now.
