I'll try to ask this question in a form of a hypothetical: I have 10 different advertising spaces and I want to rank them according to their conversion rate (conversions/views). BUT, I have an uneven amount of data on each of them. example (sorry for the formatting):

Ad_space, conversion_rate, #_of_views
1 3% 500
2 3% 15,000
3 2.9% 20,000
4 2.8% 100,000
5 2.5% 10,000
6 2.4% 5,000
7 2% 10,000
8 2% 15,000
9 1.5% 10,000
10 1% 10,000

The problem is deciding between ads who are doing well on a small sample and ads who are doing slightly less well on a bigger sample (i.e. which one is better? space 1 or 3? 2 or 4?)

I'm struggling to come up with a metric that will account both for the conversion rate and the amount of instances at the same time. I thought of centering the data around a "neutral" conversion rate and then let the # of views push the rank up for spaces above that rank, and down for spaces below it. Does this sound like the right approach? also, how would you put that into a formula?


You can simply calculate conversion_rate,

$$ P = {n_r \over n}, $$

so that $P_1 = 0.03$, $P_2 = 0.03$, and $P_2 = 0.029$, but it's better to add a correction for total number of views. One way to correct is to add "dummy" views (e.g. $1000$), so

$$ P' = {n_r \over n + 1000 }. $$

Ads with a large number of views see their modified conversion rate alter very little from their real rate, but ads with relatively few views will see their modified percentage move considerably toward low values.

This is known as "Bayesian averaging". In effect, the ads with many views will rank higher than ads with the same percentage but fewer answers.

  • $\begingroup$ I'd argue that most people would understand "Bayesian averaging" as "Bayesian model averaging" (e.g. arxiv.org/abs/1509.08864 ) $\endgroup$ – Tim Sep 6 '17 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.