1
$\begingroup$

I would like to run the following experiment , test if there is a significant difference between 2 algorithm ,the KPI I would like to test the difference for is : revenue per single impression.

My problem that in both experiments (the test and the control) , only 1% of the impressions reports for revenue, so the results of the experiments are not normally distributed. (my sample size is big : ±700M impressions for a given experiment, test 7 control)

Can I run a t-test to compare between the 2 algorithms ? based on the assumption that the diff between any 2 impressions (test vs control) is normally distributed ? should I check this assumption first ?

$\endgroup$
4
  • $\begingroup$ I got a couple of questions: is the revenue value continous or binary (yes/no)? Are the impressions in two groups the same objects, so you can make a pair-wise comparison? $\endgroup$ Commented Sep 26, 2017 at 13:29
  • $\begingroup$ is the revenue value continuous or binary --> answer : continuous $\endgroup$
    – Anat Cohen
    Commented Sep 26, 2017 at 13:37
  • $\begingroup$ Are the impressions in two groups the same objects, so you can make a pair-wise comparison? -- > Answer : not sure what do you mean by "same object" , both the control and the test have 700M impressions (the experiments have the same sample size) and the impressions where collected at the same time , meaning if in time = t0 1 impression went to the control also another impression will go to the tested experiment $\endgroup$
    – Anat Cohen
    Commented Sep 26, 2017 at 13:40
  • $\begingroup$ OK, so I assume you do not have paired samples, since I read your words as having different impressions in control and test groups. As David Ernst replied to you, you ought not care about the normality of the delta between random variables in both groups. What you have to care about is the normality of sampled Mean Statistic, which should be Gaussian for large samples (like yours) due to the Central Limit Theorem. You can run t-test, and for a double check run Mann-Whitney U test (aka Wilcoxon test) which is robust against non normality. $\endgroup$ Commented Sep 26, 2017 at 14:03

1 Answer 1

1
$\begingroup$

You don't need your sample points to be normally distributed if your sample is big enough. 30 serves as a rule of thumb. You have 1% of 700M in each group, thus twice 7M. No worries there.

As you don't have paired data on the user level, you certainly don't need to reason in terms of distribution of the difference of observations. You cannot compute such a difference anyway.

If the revenue is always the same amount for a clicking user, which you imply is not the case, then you would not compare on a per user basis but daily conversion rates. In this case you could compare daily conversion rates of both groups and pair the daily observations for better statistical power. But if you have data on a per user level for varying revenue, then don't do this.

There are other assumptions to t-tests. Equality of variances between both groups is necessary especially when the sizes of both groups are very different. But this can be dealt with by simply using Welch's version of the t-test. (Don't do Levene tests etc. just go with Welch and don't worry about variances.)

You also need i.i.d samples. Verify that your millions of sample points are independent and not some form of pseudo-replication. As you seem to be talking about individual website users, you are less likely to have pseudo replication. There can still be some if you have very many impressions per human user. If your users go onto the website many times, they will conceivably be much less likely to generate you advertising revenue when coming the 25th time in a short interval compared to the first time. Even more so if some visitors are bots and other crawlers. But as only those who click constitute your sample points, this should mitigate this problem. Have some precautions that your clicking impressions are not generated by bots and it should be fine.

Report not only p-values of the t-test but confidence intervals. P-values will almost certainly be super significant with such a large $n$, but that doesn't mean the effect is practically big enough to care about. A confidence interval tells you if your effect is big enough to care about. Because of your enormous $n$, I would do 99.9% confidence intervals, they should still be narrow.

Another problem you may need to deal with is the fact that only 1% report revenue per impression. Are you sure those 1% are representative of the remaining 99%. There will most likely be systematic effects that account for which cases are likely to be the 1%, thus systematic differences between your sample and your population (if you define your population to generalize onto like this).

Since in your case you have in both groups only 1% that clicks, it may be save to assume that in both groups the reasons for clicking or not clicking are the same. So you can compare the clickers to the other clickers. Just don't extrapolate from clickers to non clickers.

Additional assumptions I made in this answer:

  • There are no other confounding variables. Both groups of 7M are internally homogeneous. This is a very strong assumption. There are almost surely other variables to take into account than group membership (date related, geography related, gender of the user etc.) If you want to take them into account, you will be looking at an ANOVA or a regression, not t-tests.
  • You look at the data only once it has accumulated. If you don't, you need to correct your p-values for multiple testing. With 7M sample points though, this will not hurt you.
$\endgroup$
5
  • $\begingroup$ * it's 700M in each group (test vs the control) $\endgroup$
    – Anat Cohen
    Commented Sep 26, 2017 at 13:41
  • $\begingroup$ * only on 1% of the impression users tend to click , and only when a click event happened then we charge for a cost $\endgroup$
    – Anat Cohen
    Commented Sep 26, 2017 at 13:42
  • $\begingroup$ Don't do anything that assumes the 1% are identical to the 99%. Otherwise you are fine comparing the 7M that saw one website and clicked and the other 7M that saw another website and clicked. $\endgroup$ Commented Sep 26, 2017 at 13:56
  • $\begingroup$ so it's okay to run a t-test based on my user case ? not sure I am 100% understand $\endgroup$
    – Anat Cohen
    Commented Sep 26, 2017 at 13:58
  • $\begingroup$ Broadly yes, caveats are described in my updated answer. If you were to compare data continuously as it flows in instead of in a big batch like you do, additional precautions for multiple testing need to be taken. $\endgroup$ Commented Sep 26, 2017 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.