1
$\begingroup$

I'm trying to A/B test (using a t-test) if a test variation leads to an increase in time spent on the website. Should I:

  1. Method 1: Treat time spent by a user on two different days as two different data points or Method
  2. Method 2: Average them out and treat them as a single data point? (A user stays in the same group on all days)

In Method 1, my data (illustrative) would like this:
User,Day,TimeSpent,Variation
U1,D1,12,V1(Test)
U1,D2,14,V1(Test)
U2,D1,11,V2(Control)

And my sample size for V1 would be 2 and for V2 it would be 1

In Method 2, my data would like this:
User,TimeSpent,Variation
U1,13,V1(Test)
U2,11,V2(Control)

And my sample size for V1(Test) would be 1 and for V2(Control) also it would be 1.

I'm trying to identify if variation one causes a user to spend more time on the website per day.

(P.S. the data is illustrative. I will have thousands of users)

$\endgroup$
7
  • $\begingroup$ Can you be more clear? What is a variation? Will you really have sample size of only 1 and 2? Why do you list only one line for user 2? $\endgroup$
    – Joel W.
    Jan 31, 2018 at 21:26
  • $\begingroup$ Method 1 violates the independence assumption. From the t-test's point of view there are 3 unique contributors, not 2. $\endgroup$
    – HEITZ
    Jan 31, 2018 at 22:16
  • $\begingroup$ @JoelW., A variation is a change on the website whose effect I want to test. My sample size is in thousands (the data in question is for illustrative purposes). User 2 did not visit the website on day 2 hence only one line. $\endgroup$ Feb 1, 2018 at 11:00
  • $\begingroup$ @HEITZ, can you provide a reference for "independence assumption". Does the requirement for independence come from a requirement for normality? $\endgroup$ Feb 1, 2018 at 11:11
  • $\begingroup$ The situation you wish to test is still not clear to me. Why would people visit the site more than once? Will people get one of 2 versions randomly? Will the time on the site on one visit be affected by the time on the site on the previous visit? $\endgroup$
    – Joel W.
    Feb 1, 2018 at 14:57

1 Answer 1

1
$\begingroup$

T-Test assumes samples are independent of each other.
In the first case where each sample is identified by a user-date tuple, you might be violating this assumption.
For example imagine the hypotheses is:
whether a one-time free pass to normally only premium content leads to improvement in user engagement.
For this hypothesis testing treating user-date as independent samples is wrong, since the treatment(shock/variant) only affect the first visit. Notice, that in specific case, the averaging method you purposed can result in misleading results as well.

A possible practice might be - using only the first visit of each user. This should work if:
1) You have enough users.
2) You have no reason to believe any of the variants has long-term cumulative effects on users behavior.

$\endgroup$
1
  • $\begingroup$ Thanks, very helpful. Can you help me understand why averaging could be misleading? $\endgroup$ Feb 1, 2018 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.