2
$\begingroup$

Background

In business industry, I came across two different type of CTR metrics to measure our products. I want to perform hypothesis testing(AB test) for these metrics. However, I am not sure about my test statistics' distribution.

Click-through rate (CTR) is the ratio of users who click on a specific link to the number of total users who view a page, email, or advertisement. It is commonly used to measure the success of an online advertising campaign for a particular website as well as the effectiveness of email campaigns. -- wiki

The general formular of CTR is $ctr = \frac{\mathrm{the \ number \ of\ click-throughs}}{\mathrm{the \ number\ of\ exposures}}$. It can be calculated over each sample point or over the whole sample.

My Question

Let's set a click through rate scenario. Suppose we have only two user on CrossValidated landing page. I opened it 4 times and clicked it 2 time while you opened it 7 times and clicked it 3 times.

  • One metric is uPVCTR: Average of each user's PVCTR. Mine PVCTR is $\frac{2}{4}= 0.5 $ and yours is $\frac{3}{7}$. $\mathrm{uPVCTR}= \frac{\frac{2}{4} + \frac{3}{7}}{2}$

  • The other is overall PVCTR: PVCTR = $\frac{\sum{clicks}}{\sum{exposure}} = \frac{2+3}{4+7}$.

Suppose each user is independent, thus each ones' PVCTR is independent. The distribution of uPVCTR is asyptotic normal. If we have control and treatment group, and we denote $\mathrm{diff} = \mathrm{uPVCTR_{treatment}} - \mathrm{uPVCTR_{control}}$. We can use $\frac{diff}{\mathrm{var}(diff)}$ to perform a two sample T-test.

However, what's the distribution of Overall PVCTR? How to choose a test statistics for it?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You can clarify the question much by giving the general formulas for the CTR metrics. Now your formulas contain fractions with constants. Also 2-3 sentences context about CTR will give this forum a clearer view on your question. $\endgroup$
    – Match Maker EE
    Aug 28, 2020 at 6:48
  • $\begingroup$ Edited, adding explanation for CTR. This is a general statistic problem $\endgroup$
    – Travis
    Aug 28, 2020 at 8:40

1 Answer 1

Reset to default
-1
$\begingroup$

It is clear that applying the log function to $CTR$ yields a variable which is likely to be normally distributed.

Define

$ \log(CTR)=\log(\# \; click\, throughs) - \log(\# \; exposures) $

Still, $\log(CTR)$ is expected to vary with the hour, and over the weekdays.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.