Ranking based on binomial data (example: website conversions)

Question

Let's say I have a database of all websites and their corresponding viewers and buyers. I want to rank the websites based on the conversion rate or where we see more conversion.

So example:

website   viewer   buyer
abc.com   1000     10
xyz.com   10       2
def.com   200      1

How do I rank the websites so that I know the top websites are the ones which have higher conversion rates.

Conversion is buyer/viewer

Answer 1

A "dumb" ranking buy buyer/viewer is generally not what one wants, because then the websites with 1 buyer / 1 viewer are ranked to the top. To account for that, one can use the lower end of the credible interval of the unknown conversion-rate.

Explanation

The observed number of trials (viewers) and successful trials (buyers) can be produced by differing true unknown conversion-rates. The less trials we have, the broader the range of possible true unknown conversion-rates is. The credible (or confidence) interval describes this range up to a desired level of precision.

If we take for example the lower bound of the 95 % interval (which is the (100-95)/2=2.5 percentile), then the (bayesian) interpretation is: Given the data we have, the probability that the true conversion-rate is less than this bound is less 2.5 %.

Regarding the method I prefer the inverse of the cumulative beta distribution (since the beta distribution models the true unknown conversion rate in a bayesian way), but more methods are available. See Binomial proportion confidence interval wikipedia page for an overview and the r-package binom for an implementation of various methods.

Application

The inverse of the beta cumulative distribution function can be calculated by using wolfram alpha.

InverseBetaRegularized[0.025,successfultrials+1,trials-successfultrials+1]

So ...

    website   viewer   buyer  old-rate    new-rate     rank
    abc.com   1000     10     0.01         0.0055       2
    xyz.com   10       2      0.2          0.0602       1
    def.com   200      1      0.005        0.0012       3

Is there more one can do ?

One can add a so called prior. The prior consists of two parts

• prior rate: The knowledge we have without looking at a specific shop. In general taking the average across all instances is helpful
• prior weight: In your case the number of viewers. If the weight is equal to the viewers of the specific shop, then both the prior rate and the actual rate is treated equally. As soon as a shop generates more viewers, the reliability of its own estimate increases and it shifts more and more away from the prior rate.

Unfortunately, setting the weight is tricky: One approach I found helpful is to estimate the minimum amount of views a site must generate (on average) to generate a desired minimum amount of buyers (on average) given the prior rate.

The resulting ranking is calculated by $\frac{buyers + priorrate*priorweight}{viewers + priorweight}$

Applied on your example:

• The macro-average of conversion-rates ((mean(c(10/1000,2/10,1/200))) is 7.17 % = prior rate

• Let's say we need at least 3 buyers, otherwise the site is not interesting. This (3/mean(c(10/1000,2/10,1/200))) leads to ~ 42 Views = prior weight (rounded up)

  website   viewer   buyer  old-rate    new-rate     rank
  abc.com   1000     10     0.01         0.0125       3
  xyz.com   10       2      0.2          0.0963       1
  def.com   200      1      0.005        0.0166       2
  

This approach can be combined with the confidence / credible interval above by first adding a prior and then estimating the lower bound of the confidence interval.

Update 23.06.

Regarding Thomson Sampling

I had to look it up:

• https://en.wikipedia.org/wiki/Thompson_sampling
• http://jmlr.org/proceedings/papers/v23/agrawal12/agrawal12.pdf

The application makes only sense here if one has to select a particular page to present it to a user while maximizing the expected rewards (conversions). But this is only my view, maybe your frequentist friend had something different in mind.

Keywords: Reinforcement Learning, Mulitarmed Bandit

Regarding the sorting per p-value (by comparison with overall conversion-rate)

Yes, this approach makes sense. But since p is the objective probability that one would observe the current results assuming the Nullhypothesis (both are equal) is true, you may run into a problem when you have a lot of sites and two sites are way better than the average, so both of there p-value is 0. How do you sort these ?

The Bayesian approach overcomes this issue. Another option here is to calculate the pairwise subjective probability p(siteA > siteB| data) instead of using the comparison by mean-conversion-rate. This approach also allows the usage of priors. This link might be helpful:

• Bayesian AB testing

Do not get confused by the Beta-Distribution. It is just the distribution of the true unknown conversion-rate given the current data.

Last but not least: The t-test is an unnecessary approximation here, since we are dealing with a binomial distribution. See my answer to this question: Difference between G-test and t-test and which should be used for A/B testing?

Answer 2

You can make a score by doing an average. And since the values are not in the same unit, you can use a geometric mean that will make them compatible by raising them to higher dimension. We got 2 components here, 1 is the rate of conversion and second is the amount of buyers. You simply multiply the factors (or divide by the ones where the smaller value is the better). Since you're interested in ranking, then no need to compute the negative power. The multiplication is enough.

score = buyer / viewer x buyer = b^2 / v

Higher score will mean higher ranking. You could even take the log of the scores for easier reading because the log pulls back the too large and too small values to better interval.