Suppose you are an advertisement business, and you have two revenue streams
y. Let's say the value of
y is not trivially comparable - maybe they have different growth potentials. In any case though, the more the better.
Now, suppose I have two algorithms
B, and thanks to an online A/B test I know
(x=10±1, y=10±1), while
(x=15±1, y=10±1). Clearly, I should accept
B as it gives me better results.
The problem is, what if I get
A -> (x=10±1, y=10±1) and
B -> (x=15±1, y=8±1)? Surprisingly many (almost everyone) at my company claims I should "improve one, while not harming the other". But this doesn't make sense to me for two reasons:
- I introduce an arbitrary line - I essentially declare that reducing
yby however little has infinitely large negative impact, and no amount of improvement in the other KPI can compensate for it. But those values are arbitrary - if I happened to get
A -> (x=10±1, y=11±1)for whatever reason, I would have insisted on at least getting
y=11±1instead of being content with
y=10±1, which would be irrational.
- I can "sneak in" the same change if I split it in steps. Since I can't measure the result with arbitrary precision, I have to declare some Minimum Detectable Effect. I.e. at most I can say "improve one KPI while not reducing the other by more than z%". If I had an algorithm improvement that reduces
yby more than
z%I couldn't ship it, but if I hack that same change into multiple pieces, such that each step only degrades
yby less than
z%, eventually I can ship the same change!
To me it just sound illogical to say "improve x while not harming y", but I hear it so often from so many people - people that have data science background as well - that I'm thinking maybe I'm missing something here.
For me, it will make a lot more sense to estimate the relationship between
y somehow (e.g. by saying
2 * x is roughly equal to 1 * y), and say "let's improve
x + 2 * y".