# Ranking based on binomial data (example: website conversions)

Let's say I have a database of all websites and their corresponding viewers and buyers. I want to rank the websites based on the conversion rate or where we see more conversion.

So example:

website   viewer   buyer
abc.com   1000     10
xyz.com   10       2
def.com   200      1


How do I rank the websites so that I know the top websites are the ones which have higher conversion rates.

• Is this a programming question? Jun 17, 2015 at 22:42
• This is more of a statistics question. I didnt want to mention anything in the beginning and wanted to check the approach people suggest. But apparently there would be some statistical principles which would get used here. And lets just say you are given this question. How would you solve it. Forget the programming part. That anyone can code. What approach to be taken to rank the website if all we have the above data Jun 17, 2015 at 23:08
• and the conversion column is calculated as buyer/viewer? Can you provide some more explanation as to whether you are looking for suggestions for more variables to include or how to compare conversion values that come from different dataset sizes, e.g. 1000 for abc.com, 10 for xyz.com Jun 18, 2015 at 8:25
• Yes. All we have is that table. So one needs to rank the sites based on the conversion rates defined above. And yes you are right the sample is diff for different sites. so how to rank? Jun 18, 2015 at 17:48

A "dumb" ranking buy buyer/viewer is generally not what one wants, because then the websites with 1 buyer / 1 viewer are ranked to the top. To account for that, one can use the lower end of the credible interval of the unknown conversion-rate.

Explanation

The observed number of trials (viewers) and successful trials (buyers) can be produced by differing true unknown conversion-rates. The less trials we have, the broader the range of possible true unknown conversion-rates is. The credible (or confidence) interval describes this range up to a desired level of precision.

If we take for example the lower bound of the 95 % interval (which is the (100-95)/2=2.5 percentile), then the (bayesian) interpretation is: Given the data we have, the probability that the true conversion-rate is less than this bound is less 2.5 %.

Regarding the method I prefer the inverse of the cumulative beta distribution (since the beta distribution models the true unknown conversion rate in a bayesian way), but more methods are available. See Binomial proportion confidence interval wikipedia page for an overview and the r-package binom for an implementation of various methods.

Application

The inverse of the beta cumulative distribution function can be calculated by using wolfram alpha.

InverseBetaRegularized[0.025,successfultrials+1,trials-successfultrials+1]


So ...

    website   viewer   buyer  old-rate    new-rate     rank
abc.com   1000     10     0.01         0.0055       2
xyz.com   10       2      0.2          0.0602       1
def.com   200      1      0.005        0.0012       3


Is there more one can do ?

One can add a so called prior. The prior consists of two parts

• prior rate: The knowledge we have without looking at a specific shop. In general taking the average across all instances is helpful
• prior weight: In your case the number of viewers. If the weight is equal to the viewers of the specific shop, then both the prior rate and the actual rate is treated equally. As soon as a shop generates more viewers, the reliability of its own estimate increases and it shifts more and more away from the prior rate.

Unfortunately, setting the weight is tricky: One approach I found helpful is to estimate the minimum amount of views a site must generate (on average) to generate a desired minimum amount of buyers (on average) given the prior rate.

The resulting ranking is calculated by $\frac{buyers + priorrate*priorweight}{viewers + priorweight}$

• The macro-average of conversion-rates ((mean(c(10/1000,2/10,1/200))) is 7.17 % = prior rate
• Let's say we need at least 3 buyers, otherwise the site is not interesting. This (3/mean(c(10/1000,2/10,1/200))) leads to ~ 42 Views = prior weight (rounded up)

website   viewer   buyer  old-rate    new-rate     rank
abc.com   1000     10     0.01         0.0125       3
xyz.com   10       2      0.2          0.0963       1
def.com   200      1      0.005        0.0166       2


This approach can be combined with the confidence / credible interval above by first adding a prior and then estimating the lower bound of the confidence interval.

Update 23.06.

Regarding Thomson Sampling

I had to look it up:

The application makes only sense here if one has to select a particular page to present it to a user while maximizing the expected rewards (conversions). But this is only my view, maybe your frequentist friend had something different in mind.

Keywords: Reinforcement Learning, Mulitarmed Bandit

Regarding the sorting per p-value (by comparison with overall conversion-rate)

Yes, this approach makes sense. But since p is the objective probability that one would observe the current results assuming the Nullhypothesis (both are equal) is true, you may run into a problem when you have a lot of sites and two sites are way better than the average, so both of there p-value is 0. How do you sort these ?

The Bayesian approach overcomes this issue. Another option here is to calculate the pairwise subjective probability p(siteA > siteB| data) instead of using the comparison by mean-conversion-rate. This approach also allows the usage of priors. This link might be helpful:

Do not get confused by the Beta-Distribution. It is just the distribution of the true unknown conversion-rate given the current data.

Last but not least: The t-test is an unnecessary approximation here, since we are dealing with a binomial distribution. See my answer to this question: Difference between G-test and t-test and which should be used for A/B testing?

• Very well answered. Unfortunately I knew some Bayesian thing is to be used but I am not a very Bayesian guy as of now. Couple of things: One friend of mine suggested something called as randomised prob matching using Thomson Sampling. According to him this is a classic example of Thomson Sampling. Is this correct? Secondly, the way I thought about this is and again since I know as of now only frequentist approach is that consider each site as a binomial dist. Like for abc.com we have 1000 views. so consider it as a sample of 1000 with 1 or 0 as buys. Contd.. on next comment Jun 20, 2015 at 2:29
• .... and then find conversion rate for each site as given. Then find the overall conversion rate across site. Total buys/total views. Do t-test of each site conversion rate with overall conversion rate and then rank by increasing order of p value. Lower the p value higher the significant diff between conversion rate of a site with the overall conversion rate and hence low p value. Is this approach sound? Jun 20, 2015 at 2:30
• @user Sorry for the late reply, I was away over the course of the weekend. I have updated my answer. Jun 23, 2015 at 8:18

You can make a score by doing an average. And since the values are not in the same unit, you can use a geometric mean that will make them compatible by raising them to higher dimension. We got 2 components here, 1 is the rate of conversion and second is the amount of buyers. You simply multiply the factors (or divide by the ones where the smaller value is the better). Since you're interested in ranking, then no need to compute the negative power. The multiplication is enough.