I'm not sure how to ask this question, so please forgive if I'm vague. I'm new to DS and don't speak the language yet. :)
Business process A (bpA) assigns accounts to handlers based on existing logic. Business process B (bpB) uses ML to assign accounts based on better logic, but I'm not sure how much better.
I have a pool of data stretching back a few years, which I take as my baseline. This is entirely based on bpA, the "old" way. I train and run my model to get bpB recommendations. I look at the list of recommendations and ask, "where did bpA do what bpB recommended?" That's my first bucket. Everything else, where we did NOT do what bpB recommended, get tossed into the second bucket. Then I look at the average success rate -- bucket 1 might have an average success rate of 75%, while bucket 2 might have an average success rate of 65%.
The easy answer is to say, "10% improvement". I suspect that this is not the correct answer, though.
Where should I be looking? What should I be reading? Thanks!