T-test : Average Time Spent on Website

Question

I'm trying to A/B test (using a t-test) if a test variation leads to an increase in time spent on the website. Should I:

1. Method 1: Treat time spent by a user on two different days as two different data points or Method
2. Method 2: Average them out and treat them as a single data point? (A user stays in the same group on all days)

In Method 1, my data (illustrative) would like this:
User,Day,TimeSpent,Variation
U1,D1,12,V1(Test)
U1,D2,14,V1(Test)
U2,D1,11,V2(Control)

And my sample size for V1 would be 2 and for V2 it would be 1

In Method 2, my data would like this:
User,TimeSpent,Variation
U1,13,V1(Test)
U2,11,V2(Control)

And my sample size for V1(Test) would be 1 and for V2(Control) also it would be 1.

I'm trying to identify if variation one causes a user to spend more time on the website per day.

(P.S. the data is illustrative. I will have thousands of users)

Answer 1

T-Test assumes samples are independent of each other.
In the first case where each sample is identified by a user-date tuple, you might be violating this assumption.
For example imagine the hypotheses is:
whether a one-time free pass to normally only premium content leads to improvement in user engagement.
For this hypothesis testing treating user-date as independent samples is wrong, since the treatment(shock/variant) only affect the first visit. Notice, that in specific case, the averaging method you purposed can result in misleading results as well.

A possible practice might be - using only the first visit of each user. This should work if:
1) You have enough users.
2) You have no reason to believe any of the variants has long-term cumulative effects on users behavior.