What is the general right call to make from a marginal difference in A/B test results in recommender system? This was one of the business-related questions from my technical interview last week for a data science position in a recommender system team at a search engine company focusing on advertisement banners. I'm curious what the general response for this question should be.
Problem: We have two models for the item recommender system, X and Y, where X is a model that the company has been using for years and Y is a model that has recently been developed and suggested. Given that we have performed an A/B test, and if Y returns a very marginally better evaluation metric compared to X with an extremely low p-value, would it be optimal to make a complete switch to model Y?
Clarification: I asked a couple of questions.

*

*What was the proportion of samples for models X and Y?

*What was the evaluation metric?

*How long was the duration of the A/B test?

The answers to those clarifications are that: the proportion of samples and the type of the evaluation metric are irrelevant, and the duration of the test was a few weeks.
My Answer: I have verbally processed my answer for a couple minutes, in which I won't write them down verbatim, but the essence of my answer was that:
"Overthrowing a working model due to a single A/B test result is not a business-optimal conclusion. We should first of all, check whether the A/B test samples represented the essential trend and entirety of the data. For example, if there is a monthly effect to the item purchasing trend, a few weeks worth of A/B test result may not be representative."

Any additional inputs or new ideas are appreciated.
 A: You should consider the cost of making the full production shift from X to Y. Say the marginal improvement only results in increased revenue of \$1000 per month, but the cost to do the full switch is $1,000,000. It will take 1000 months to break even, which probably does not make sense to switch then. (But if costless, e.g. just change underlying predictive model and end user outcomes/infrastructure are all the same, then yeah go for the marginal increased revenue.)
You can critique whatever model forever, so if you want to be ultra-risk averse you could always say 'we should not migrate' due to model uncertainty. E.g. you could say 'data in 2022 may not be relevant to behavior in 2023' ad-infinitum.
I suspect they wanted you to understand how to translate the model to actionable decision making. Which involves calculating costs/benefits. Those costs/benefits may have distributions (so not totally outside the domain of stats), but is somewhat of a different task than traditional stats. By saying "extremely low p-value" I presume they want you to ignore uncertainty in the point estimate, and just focus on the economic/utility estimate for whether it is worth it to make the switch.
