I'm not sure how to ask this question, so please forgive if I'm vague. I'm new to DS and don't speak the language yet. :)

Business process A (bpA) assigns accounts to handlers based on existing logic. Business process B (bpB) uses ML to assign accounts based on better logic, but I'm not sure how much better.

I have a pool of data stretching back a few years, which I take as my baseline. This is entirely based on bpA, the "old" way. I train and run my model to get bpB recommendations. I look at the list of recommendations and ask, "where did bpA do what bpB recommended?" That's my first bucket. Everything else, where we did NOT do what bpB recommended, get tossed into the second bucket. Then I look at the average success rate -- bucket 1 might have an average success rate of 75%, while bucket 2 might have an average success rate of 65%.

The easy answer is to say, "10% improvement". I suspect that this is not the correct answer, though.

Where should I be looking? What should I be reading? Thanks!

  • 1
    $\begingroup$ Could you be more specific about what you have done? Have you the capacity to assign the same account to different handlers (i.e. Do you have paired data for bpA and bpB)? How do you measure performance after assignment? $\endgroup$ – ReneBt Sep 8 '18 at 6:52
  • $\begingroup$ Yes, the account can be assigned to different handlers. The role of the ML "engine" is to determine which handler is likely to have greatest success with a given account. Imagine this is a large ticketing system -- which employee is likely to resolve the customer's needs most quickly / efficiently / profitably? We measure performance based on "what percent of request was met?". It's easy to evaluate performance for each account in our business, I just don't know how to quantify the optimization accurately. $\endgroup$ – Duncan Sep 11 '18 at 15:18
  • $\begingroup$ @Duncan Please consider accepting an answer. If you find no answer satisfactory, please consider editing your question(s) to provide more information. If you want to motivate answerers, please consider starting a bounty. Accepting an answer shows your appreciation, rewards the author, provides incentive to others and informs everyone that your issue is resolved. You can always change your mind and accept a different answer later on. $\endgroup$ – marianoju Sep 17 '18 at 15:43

I then compare performance rates between "bpA did what bpB recommended" and the inverse. I see a difference in performance between the two sets, but I am not sure that the number I see is the number I should be reporting.

If you see a difference in performance you need to establish if it is statistically significant.

  • Have you measured the accuracy of your model?
  • Can you explain how you measure performance?

It would be helpful if you provided some Minimal, Complete, and Verifiable example.

  • $\begingroup$ How we measure performance: we use a percentage. Given the properties of an account, one handler might eventually score 78% and another might eventually score 85%, so we want to send the account to the handler expected to score highest. $\endgroup$ – Duncan Sep 11 '18 at 15:36
  • $\begingroup$ @Duncan I was not asking for the format of your results (percentages) but for your definition of performance (e.g. ROI?) and methodology. Your results are represented in percentages of what exactly? $\endgroup$ – marianoju Sep 13 '18 at 5:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.