# How valid is saying "improve x while not decreasing y" in business metric optimization?

Suppose you are an advertisement business, and you have two revenue streams x and y. Let's say the value of x and y is not trivially comparable - maybe they have different growth potentials. In any case though, the more the better.

Now, suppose I have two algorithms A and B, and thanks to an online A/B test I know A yields (x=10±1, y=10±1), while B yields (x=15±1, y=10±1). Clearly, I should accept B as it gives me better results.

The problem is, what if I get A -> (x=10±1, y=10±1) and B -> (x=15±1, y=8±1)? Surprisingly many (almost everyone) at my company claims I should "improve one, while not harming the other". But this doesn't make sense to me for two reasons:

1. I introduce an arbitrary line - I essentially declare that reducing x or y by however little has infinitely large negative impact, and no amount of improvement in the other KPI can compensate for it. But those values are arbitrary - if I happened to get A -> (x=10±1, y=11±1) for whatever reason, I would have insisted on at least getting y=11±1 instead of being content with y=10±1, which would be irrational.
2. I can "sneak in" the same change if I split it in steps. Since I can't measure the result with arbitrary precision, I have to declare some Minimum Detectable Effect. I.e. at most I can say "improve one KPI while not reducing the other by more than z%". If I had an algorithm improvement that reduces y by more than z% I couldn't ship it, but if I hack that same change into multiple pieces, such that each step only degrades y by less than z%, eventually I can ship the same change!

To me it just sound illogical to say "improve x while not harming y", but I hear it so often from so many people - people that have data science background as well - that I'm thinking maybe I'm missing something here.

For me, it will make a lot more sense to estimate the relationship between x and y somehow (e.g. by saying 2 * x is roughly equal to 1 * y), and say "let's improve x + 2 * y".

• If you can assign values to x and y that are explicit and on the same scale--e.g., dollars--then you can solve this algebraically and show how a scenario in which one stream suffers might still be optimal. But I suspect cultural aspects will still make that a tough sell... Feb 5, 2020 at 22:41