I am working on a binary classification problem with 75:25 class portion for 1000 records. Objective is to find out whethet the supplier met the target or not.
Currently my recall for minority class stands at 40%. Meaning, it identifies 22 out of 55 actual negatives correctly.
Now, my business wants go live with this performance and they also say that they will consider this project/model a success if in real time, at least 1 out of 22 (identified by model) as less likley to meet the target (converts to meeting the target upon follow up by our sales team). So, basically if we even get 1000$ from the suppliers (identified as less likley to meet the target) after follow up, we consider this model a success (because it results in revenue for the company. Otherwise, our sales team may not know who to follow up etc).
My concern is as below
a) Even if my recall had been 90%, do you think it is possible to hold model responsible for ROI? Meaning, whether there is increase in revenue or no revenue at all, should model be credited for it?
Because, I feel model can only provide some strategy as to who to follow up, identifying the right suppliers etc. But do you think model can be deemed as success in real life only if it generates revenue? I feel the generation/non generation of revenue could also be due to the skills of sales team, supplier suddem demand etc.
I feel model assessment is over once we assess the performance on test data? Am I wrong to think that way?
But why is model considered a success only based on the ROI? Is it the right way to assess the usefulness of model? Is it possible to seperate the role of model from other factors that influence the ROI?
So, based on your experience, how and when a model is considered a success in real world?