1
$\begingroup$

Suppose you are an advertisement business, and you have two revenue streams x and y. Let's say the value of x and y is not trivially comparable - maybe they have different growth potentials. In any case though, the more the better.

Now, suppose I have two algorithms A and B, and thanks to an online A/B test I know A yields (x=10±1, y=10±1), while B yields (x=15±1, y=10±1). Clearly, I should accept B as it gives me better results.

The problem is, what if I get A -> (x=10±1, y=10±1) and B -> (x=15±1, y=8±1)? Surprisingly many (almost everyone) at my company claims I should "improve one, while not harming the other". But this doesn't make sense to me for two reasons:

  1. I introduce an arbitrary line - I essentially declare that reducing x or y by however little has infinitely large negative impact, and no amount of improvement in the other KPI can compensate for it. But those values are arbitrary - if I happened to get A -> (x=10±1, y=11±1) for whatever reason, I would have insisted on at least getting y=11±1 instead of being content with y=10±1, which would be irrational.
  2. I can "sneak in" the same change if I split it in steps. Since I can't measure the result with arbitrary precision, I have to declare some Minimum Detectable Effect. I.e. at most I can say "improve one KPI while not reducing the other by more than z%". If I had an algorithm improvement that reduces y by more than z% I couldn't ship it, but if I hack that same change into multiple pieces, such that each step only degrades y by less than z%, eventually I can ship the same change!

To me it just sound illogical to say "improve x while not harming y", but I hear it so often from so many people - people that have data science background as well - that I'm thinking maybe I'm missing something here.

For me, it will make a lot more sense to estimate the relationship between x and y somehow (e.g. by saying 2 * x is roughly equal to 1 * y), and say "let's improve x + 2 * y".

Improve this question
$\endgroup$
1
  • $\begingroup$ If you can assign values to x and y that are explicit and on the same scale--e.g., dollars--then you can solve this algebraically and show how a scenario in which one stream suffers might still be optimal. But I suspect cultural aspects will still make that a tough sell... $\endgroup$
    – ulfelder
    Feb 5, 2020 at 22:41

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.