1
$\begingroup$

I would like to run a t-Test (Two sampling assuming equal variances) in order to evaluate the lift observed in an A\B test of a mobile ad campaign. My main KPI is Revenue per 1000 views.
My data consist of the following dimensions: Date, Publisher, Group_name (the variation of the test) .
I have two questions:
1. Is it correct to have my t-test run on an aggregated pivot of the data, while calculating the average KPI per day? for example:

date A B
2019-02-23 2.64 3.37
2019-02-24 2.89 3.92
2019-02-25 3.70 5.21
2019-02-26 3.57 4.06
2019-02-27 2.78 3.86
2019-02-28 3.12 3.57
AVERAGE of RPM 3.12 4.00
  1. Considering the underlying data has also Publisher level of details, I'm trying to figure out what Average should I use in the pivot - Weighted Average or Simple Average. Following the above example of the simple average, here's the pivot of weighted average:

date A B
2019-02-23 1.41 1.71
2019-02-24 1.27 1.54
2019-02-25 1.31 1.63
2019-02-26 1.46 1.58
2019-02-27 1.35 1.71
2019-02-28 2.15 2.33
AVERAGE of RPM 1.49 1.74


Which one, if any, is the correct method?
Thanks

EDIT:
More details regarding my design:
My product is a mobile widget which shows recommendations for other content on the internet, which could be either organic or promoted content. The Widget is embedded at the bottom of several publishers sites, and I generate revenue when customers click on any of the content.
I have two possible setups for the widget - the current (Group A) - shows 1 organic recommendation and 2 promoted recommendations; and the new setup (Group B) shows 2 promoted recommendations and then 1 organic recommendation.
I now run an experiment and splitting my traffic equally and randomly between the two groups for 7 days.
After concluding the experiment, I'm calculating my Revenue per 1,000 page views (my KPI), and I noticed a lift of 17% in favor of Group B.
Trying to statistically test my results, I'm trying to run a Two-Sample Assuming Equal Variances T-test in Excel.
Question is, What array of values should I input to the excel in order to get my results?
I now understand that aggregating the results by date is probably not the way, however, I cannot just test "Mean A" vs. "Mean B" in this test.
My raw data looks like this: | date | publisher_id | platform | group_name | page_views | revenue | Revenue_per_1k_page_views | |------------|--------------|----------|------------|------------|---------|---------------------------| | 2019-02-23 | 106 | Mobile | A | 30,494 | 82.93 | 2.71 | | 2019-02-23 | 106 | Mobile | B | 30,079 | 100.82 | 3.35 | | 2019-02-24 | 106 | Mobile | A | 24,098 | 87.79 | 3.64 | | 2019-02-24 | 106 | Mobile | B | 22,846 | 100.00 | 4.37 |
enter image description here

$\endgroup$
6
  • $\begingroup$ Why would you compute the KPI per day? Is there good reason to believe the effect is heterogenous between groups across days of the week? $\endgroup$ Jul 30, 2021 at 22:18
  • $\begingroup$ I thought computing the KPI across days is necessary in order to plot the 2 distributions and calculate the variance. $\endgroup$
    – Asaf Lahav
    Jul 31, 2021 at 4:06
  • $\begingroup$ The distribution and variance of what? If the AB test was properly randomized, and the intervention did not explicitly try to modulate the KPI on day of week, then doing this results in lower precision estimates. Can you explain the experimental design a little more? WHat was the intervention? $\endgroup$ Jul 31, 2021 at 11:10
  • $\begingroup$ Thanks so much for your support. I have edited my post with further details. hope this clarifies my question. $\endgroup$
    – Asaf Lahav
    Jul 31, 2021 at 18:46
  • $\begingroup$ do you only have information in this aggregated form? $\endgroup$ Aug 1, 2021 at 3:27

2 Answers 2

4
$\begingroup$

I'm going to get a little mathy here.

Let $y_{x, j, i}$ be the $i^{th}$ amount of revenue earned from a single click from a user in group $x$ on the $j$th day of the experiment. Your data presently consist of the quantities

$$ R_{x, j} = \sum_i^{n_{x,j}} y_{x, j.i}$$

where $n_{x,j}$ is the size of each arm on each day. Assuming the $y$ are iid, this we can apply the central limit theorem.

$$ R_{x,j} \sim \mathcal{N}(n_{x,j} \mu_x, n_{x, j}\sigma^2) $$

Where I have assumed here that only the means differ by group and not the variance.

We can use the revenue in a linear model to estimate the effect of the exposure in the following way. Let $z$ be a binary indicator variable for exposure to the treatment. The means can be modelled as

$$ R_{x, j}/n_{x, j} = \beta_0 + \beta_1 z + \epsilon $$

and weighted by the number of views. This can be done in R via


z = c(0, 1, 0, 1)
n = c(30494, 30079, 24098, 22846)
rv = c(82.93, 100.82, 87.79, 100)/n
model = lm(rv~z, weights = n)
summary(model)
summary(model)

Call:
lm(formula = rv ~ z, weights = n)

Weighted Residuals:
       1        2        3        4 
-0.07119 -0.07676  0.08008  0.08808 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)  
(Intercept) 0.0031272  0.0004797   6.518   0.0227 *
z           0.0006672  0.0006838   0.976   0.4321  
---
Signif. codes:  
0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1121 on 2 degrees of freedom
Multiple R-squared:  0.3225,    Adjusted R-squared:  -0.0162 
F-statistic: 0.9522 on 1 and 2 DF,  p-value: 0.4321
```
$\endgroup$
3
  • $\begingroup$ Is there supposed to be a difference in the indexing of $y_x,j,i$ and $y_x,i,j$ or is it just a typo? $\endgroup$ Aug 1, 2021 at 22:22
  • $\begingroup$ @user1916067 Typo, corrected thanks $\endgroup$ Aug 2, 2021 at 2:58
  • $\begingroup$ Thanks Demetri! $\endgroup$
    – Asaf Lahav
    Aug 3, 2021 at 3:10
1
$\begingroup$

If the data you are showing in the table is all your data, I would suggest a simple approach - your KPI is higher for option B in every single group, so it is obviously better.

This is informally, a non-parametric test. If Group A and B were the same, you would expect A to win half the time and B to win half the time. In this case, B won all 6 times. This is equivalent to saying that you flipped a coin 6 times and got all 6 heads.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.