8
$\begingroup$

I need to track the conversion rate of a landing page. I want to figure out how many clicks I need to get before I can be sure that I have an accurate conversion rate calculation.

I first figured I would calculate the confidence interval using the appropriate formula for a proportion, and find the number of clicks needed to make the interval sufficiently tight. However, according to this link you need at least 1,000 clicks before you can assume the distribution is normal. Unfortunately I can't get that many clicks for all of my landing pages. (I'm using landing pages as a parallel for what I'm really doing - in reality I'm testing hundreds of CPA advertisements for their conversion rates which prevents me from giving each thousands of clicks. This detail doesn't change the problem, though, so it can be ignored).

I then had a different idea - I'd go through the click/conversion data that I do have so far for previous landing pages using different "window sizes", such as a window of 10 clicks, 50 clicks, 100 clicks, 200 clicks, etc. For each window size, I'd go through each click in sequential order and calculate the conversion rate using the next x clicks, starting from the one I'm inspecting. I'd then calculate the variance of the conversion rates I found. I'd then find a window size that gives me a desired amount of minimal variance across the majority of the landing pages, and then use that as the number of clicks to test for for future tests.

Does this approach seem to make sense / has anyone heard of something similar? Probably more importantly, what is the normal way to accomplish this goal?

Thanks!

$\endgroup$

1 Answer 1

4
$\begingroup$

Paul. A conversion rate is defined as:

$\textrm{Conversion rate} = r =\frac{\textrm{Number of goal achievements}}{\textrm{Visits}} = \frac{s}{n}$.

Assuming that the number of goal achievements is basically the number of success $s$ out of the number of trials $v$ (rather than the number of events per unit time or space), then what you are trying to do with your conversion rate data is estimate the underlying but unknown probability of conversion $\kappa$. There is absolutely no need to make a normality assumption in estimating the rate or its uncertainty. Instead, you could use the Bayesian Beta-binomial model for estimating the probability distribution of an unknown proportion.

In the Beta-binomial model, your conversion rate data $v$ follows a binomial distribution with size $n$ and probability $\kappa$:

$s \sim \textrm{Bin}(n,\kappa)$

Of course, you don't know what $\kappa$ is, so you are going to use Bayes' theorem to estimate it by combining your prior beliefs about what the probability might be and your conversion rate data. It turns out that a very useful model for the distribution of your priors beliefs about the probability in this case is the Beta distribution with concentration parameters $\alpha$ and $\beta$.

$\kappa \sim \textrm{Beta}(\alpha, \beta)$

In the Beta distribution, the parameters $\alpha$ and $\beta$ represent your prior beliefs about the concentration of successes and failures. The greater one concentration parameter is relative to another, the greater your belief that the probability favors that event. Also, the greater the sum of the concentration parameters $\alpha + \beta$, the more prior information you have about the conversion rate (say, from previous experiments using the same landing page), and the more certain you are in the expected probability $\frac{\alpha}{\alpha + \beta}$.

There is a nifty mathematical result here. It turns out that the posterior distribution of the conversion probability $\kappa$ follows a Beta distribution with the concentration parameters being a simply modification of those in your prior. The posterior distribution of the conversion probability is:

$\textrm{Pr}(\kappa|s,n,\alpha,\beta) \sim \textrm{Beta}(\alpha + s, \beta + n - s)$

That is, you just add the counts in your data to the appropriate concentration parameter ($\alpha$ being the concentration of successes and $\beta$ that of failures), and voila!

But how should you set the values of $\alpha$ and $\beta$? If you have prior information about the conversion rate for that landing page, perhaps you could set the concentration parameters to be the counts of those prior experiments. But be careful: the bigger the prior sample size, the more new conversion data you will need to overwhelm your prior beliefs. Then again, you could set the parameters so that the prior expected value $\frac{\alpha}{\alpha + \beta}$ is equal to the conversion probability from your prior experiments, but choose the parameters also so that their sum is low, reflecting your lack of information about the present landing page, say, because this landing page is much different from your previous ones. How you set the prior depends on your circumstances and your beliefs and how strong those beliefs are.

Another option is to claim ignorance about what the value of $\kappa$ might be. In this case, you could set $\alpha = \beta = 1$, which is equivalent to a continuous uniform (i.e., flat) prior distribution. Some suggest that you should use the Jeffrey's prior distribution, wherein $\alpha = \beta = 1/2$ instead, which as a U-shape with modes at 0 and 1.

Regardless of what prior distribution you choose, now you can estimate the expected value of the conversion probability, which is:

$\textrm{E}(\kappa|s,n,\alpha,\beta) = \frac{\alpha + s}{\alpha + \beta + n}$

You could also estimate the posterior variance using the formula for the Beta distribution variance, or the posteiror median, or the posteiror kurtosis, or the posterior skewness, or what have you. You could use a computer program such as R to estimate the credible range for conversion rate given your data. For example, you could estimate the posterior 95% confidence interval by firing up R and running the following code (assuming that you've defined $\alpha$, $\beta$, $s$, and $n$ in your code previously):

post_CI <- qbeta(c(0.025, 0.975), alpha, beta)

You could also use simulations from the posterior distribution to compute any number of alternative credible intervals, such as the highest posterior density interval or lowest posterior loss interval. You should seriously look into highest posterior density interval because the Beta distribution can be quite skewed, causing the traditional quantile interval approach to allow values into the interval that have lower posterior probability than values that are not in the interval. Below is the code for computing the highest posterior density interval, assuming you've defined everything as before:

sims <- rbeta(1000000, alpha, beta)
require(coda) || install.packages("coda")
post_HDI <- HPDinterval(as.mcmc(sims), prob=0.95)
$\endgroup$
6
  • $\begingroup$ I must say that your suggesting here a beysian approach to some one who might be looking for a frequentist answer, the standard of the industry (which is frequentistic!) is to use the binomial distribution without the prior, see here an example r-bloggers.com/… $\endgroup$ Jul 6, 2014 at 5:39
  • 3
    $\begingroup$ I must say that I've seen plenty of people in the industry use Bayesian methods for computing conversion rates, and I also must say that I don't know what the value of $\kappa$ is, and I don't trust the maximum likelihood estimate of its expectation if I don't have that much data. Of course, if I do have lots of data, the two approaches give the same answer, so who cares? Well, I do, because it turns out at least for me that the Bayesian approach is easier to understand. To each their own I guess. $\endgroup$ Jul 6, 2014 at 5:49
  • $\begingroup$ I like your attitude, and ive been tring to use the beysian approach as well (see stats.stackexchange.com/questions/105260/…) but there's seem to be some resistance to it, out there. $\endgroup$ Jul 6, 2014 at 5:53
  • 1
    $\begingroup$ In the post you link to, I see you comparing two posterior Beta distributions that have improper priors with concentration parameters set to zero. So long as your posterior distributions are proper, that is a perfectly legitimate way to compare two proportions. In fact, I prefer it, because it doesn't rely on normal approximations. Just be careful, because if you ever got a zero count for one of your categories, your posterior would no longer be proper. $\endgroup$ Jul 6, 2014 at 6:11
  • $\begingroup$ Hi @Brash Equilibrium could you help me with this one? stats.stackexchange.com/questions/105370/… $\endgroup$ Jul 7, 2014 at 5:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.