Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I need to track the conversion rate of a landing page. I want to figure out how many clicks I need to get before I can be sure that I have an accurate conversion rate calculation.

I first figured I would calculate the confidence interval using the appropriate formula for a proportion, and find the number of clicks needed to make the interval sufficiently tight. However, according to this link you need at least 1,000 clicks before you can assume the distribution is normal. Unfortunately I can't get that many clicks for all of my landing pages. (I'm using landing pages as a parallel for what I'm really doing - in reality I'm testing hundreds of CPA advertisements for their conversion rates which prevents me from giving each thousands of clicks. This detail doesn't change the problem, though, so it can be ignored).

I then had a different idea - I'd go through the click/conversion data that I do have so far for previous landing pages using different "window sizes", such as a window of 10 clicks, 50 clicks, 100 clicks, 200 clicks, etc. For each window size, I'd go through each click in sequential order and calculate the conversion rate using the next x clicks, starting from the one I'm inspecting. I'd then calculate the variance of the conversion rates I found. I'd then find a window size that gives me a desired amount of minimal variance across the majority of the landing pages, and then use that as the number of clicks to test for for future tests.

Does this approach seem to make sense / has anyone heard of something similar? Probably more importantly, what is the normal way to accomplish this goal?

Thanks!

share|improve this question
add comment

1 Answer 1

Paul. A conversion rate is defined as:

$\textrm{Conversion rate} = r =\frac{\textrm{Number of goal achievements}}{\textrm{Visits}} = \frac{s}{n}$.

Assuming that the number of goal achievements is basically the number of success $s$ out of the number of trials $v$ (rather than the number of events per unit time or space), then what you are trying to do with your conversion rate data is estimate the underlying but unknown probability of conversion $\kappa$. There is absolutely no need to make a normality assumption in estimating the rate or its uncertainty. Instead, you could use the Bayesian Beta-binomial model for estimating the probability distribution of an unknown proportion.

In the Beta-binomial model, your conversion rate data $v$ follows a binomial distribution with size $n$ and probability $\kappa$:

$s \sim \textrm{Bin}(n,\kappa)$

Of course, you don't know what $\kappa$ is, so you are going to use Bayes' theorem to estimate it by combining your prior beliefs about what the probability might be and your conversion rate data. It turns out that a very useful model for the distribution of your priors beliefs about the probability in this case is the Beta distribution with concentration parameters $\alpha$ and $\beta$.

$\kappa \sim \textrm{Beta}(\alpha, \beta)$

In the Beta distribution, the parameters \alpha and \beta represent your prior beliefs about the concentration of successes and failures. The greater one concentration parameter is relative to another, the greater your belief that the probability favors that event. Also, the greater the sum of the concentration parameters $\alpha + \beta$, the more prior information you have about the conversion rate (say, from previous experiments using the same landing page), and the more certain you are in the expected probability $\frac{\alpha}{\alpha + \beta}$.

There is a nifty mathematical result here. It turns out that the posterior distribution of the conversion probability $\kappa$ follows a Beta distribution with the concentration parameters being a simply modification of those in your prior. The posterior distribution of the conversion probability is:

$\textrm{Pr}(\kappa|s,n,\alpha,\beta) \sim \textrm{Beta}(\alpha + s, \beta + n - s)$

That is, you just add the counts in your data to the appropriate concentration parameter (\alpha being the concentration of successes and \beta that of failures), and voila!

But how should you set the values of $\alpha$ and $\beta$? If you have prior information about the conversion rate for that landing page, perhaps you could set the concentration parameters to be the counts of those prior experiments. But be careful: the bigger the prior sample size, the more new conversion data you will need to overwhelm your prior beliefs. Then again, you could set the parameters so that the prior expected value $\frac{\alpha}{\alpha + \beta}$ is equal to the conversion probability from your prior experiments, but choose the parameters also so that their sum is low, reflecting your lack of information about the present landing page, say, because this landing page is much different from your previous ones. How you set the prior depends on your circumstances and your beliefs and how strong those beliefs are.

Another option is to claim ignorance about what the value of $\kappa$ might be. In this case, you could set $\alpha = \beta = 1$, which is equivalent to a continuous uniform (i.e., flat) prior distribution. Some suggest that you should use the Jeffrey's prior distribution, wherein $\alpha = \beta = 1/2$ instead, which as a U-shape with modes at 0 and 1.

Regardless of what prior distribution you choose, now you can estimate the expected value of the conversion probability, which is:

$\textrm{E}(\kappa|s,n,\alpha,\beta) = \frac{\alpha + s}{\alpha + \beta + n}$

You could also estimate the posterior variance using the formula for the Beta distribution variance, or the posteiror median, or the posteiror kurtosis, or the posterior skewness, or what have you. You could use a computer program such as R to estimate the credible range for conversion rate given your data. For example, you could estimate the posterior 95% confidence interval by firing up R and running the following code (assuming that you've defined $\alpha$, $\beta$, $s$, and $n$ in your code previously):

post_CI <- qbeta(c(0.025, 0.975), alpha, beta)

You could also use simulations from the posterior distribution to compute any number of alternative credible intervals, such as the highest posterior density interval or lowest posterior loss interval. You should seriously look into highest posterior density interval because the Beta distribution can be quite skewed, causing the traditional quantile interval approach to allow values into the interval that have lower posterior probability than values that are not in the interval. Below is the code for computing the highest posterior density interval, assuming you've defined everything as before:

sims <- rbeta(1000000, alpha, beta)
require(coda) || install.packages("coda")
post_HDI <- HPDinterval(as.mcmc(sims), prob=0.95)
share|improve this answer
    
I must say that your suggesting here a beysian approach to some one who might be looking for a frequentist answer, the standard of the industry (which is frequentistic!) is to use the binomial distribution without the prior, see here an example r-bloggers.com/… –  Yehoshaphat Schellekens Jul 6 at 5:39
1  
I must say that I've seen plenty of people in the industry use Bayesian methods for computing conversion rates, and I also must say that I don't know what the value of $\kappa$ is, and I don't trust the maximum likelihood estimate of its expectation if I don't have that much data. Of course, if I do have lots of data, the two approaches give the same answer, so who cares? Well, I do, because it turns out at least for me that the Bayesian approach is easier to understand. To each their own I guess. –  Brash Equilibrium Jul 6 at 5:49
    
I like your attitude, and ive been tring to use the beysian approach as well (see stats.stackexchange.com/questions/105260/…) but there's seem to be some resistance to it, out there. –  Yehoshaphat Schellekens Jul 6 at 5:53
1  
In the post you link to, I see you comparing two posterior Beta distributions that have improper priors with concentration parameters set to zero. So long as your posterior distributions are proper, that is a perfectly legitimate way to compare two proportions. In fact, I prefer it, because it doesn't rely on normal approximations. Just be careful, because if you ever got a zero count for one of your categories, your posterior would no longer be proper. –  Brash Equilibrium Jul 6 at 6:11
1  
Thanks for the help! –  Yehoshaphat Schellekens Jul 6 at 6:17
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.