2
$\begingroup$

Background

In business industry, I came across two different type of CTR metrics to measure our products. I want to perform hypothesis testing(AB test) for these metrics. However, I am not sure about my test statistics' distribution.

Click-through rate (CTR) is the ratio of users who click on a specific link to the number of total users who view a page, email, or advertisement. It is commonly used to measure the success of an online advertising campaign for a particular website as well as the effectiveness of email campaigns. -- wiki

The general formular of CTR is $ctr = \frac{\mathrm{the \ number \ of\ click-throughs}}{\mathrm{the \ number\ of\ exposures}}$. It can be calculated over each sample point or over the whole sample.

My Question

Let's set a click through rate scenario. Suppose we have only two user on CrossValidated landing page. I opened it 4 times and clicked it 2 time while you opened it 7 times and clicked it 3 times.

  • One metric is uPVCTR: Average of each user's PVCTR. Mine PVCTR is $\frac{2}{4}= 0.5 $ and yours is $\frac{3}{7}$. $\mathrm{uPVCTR}= \frac{\frac{2}{4} + \frac{3}{7}}{2}$

  • The other is overall PVCTR: PVCTR = $\frac{\sum{clicks}}{\sum{exposure}} = \frac{2+3}{4+7}$.

Suppose each user is independent, thus each ones' PVCTR is independent. The distribution of uPVCTR is asyptotic normal. If we have control and treatment group, and we denote $\mathrm{diff} = \mathrm{uPVCTR_{treatment}} - \mathrm{uPVCTR_{control}}$. We can use $\frac{diff}{\mathrm{var}(diff)}$ to perform a two sample T-test.

However, what's the distribution of Overall PVCTR? How to choose a test statistics for it?

$\endgroup$
2
  • 1
    $\begingroup$ You can clarify the question much by giving the general formulas for the CTR metrics. Now your formulas contain fractions with constants. Also 2-3 sentences context about CTR will give this forum a clearer view on your question. $\endgroup$ Aug 28 '20 at 6:48
  • $\begingroup$ Edited, adding explanation for CTR. This is a general statistic problem $\endgroup$
    – Travis
    Aug 28 '20 at 8:40
0
$\begingroup$

It is clear that applying the log function to $CTR$ yields a variable which is likely to be normally distributed.

Define

$ \log(CTR)=\log(\# \; click\, throughs) - \log(\# \; exposures) $

Still, $\log(CTR)$ is expected to vary with the hour, and over the weekdays.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.