I'm trying to A/B test (using a t-test) if a test variation leads to an increase in time spent on the website. Should I:
- Method 1: Treat time spent by a user on two different days as two different data points or Method
- Method 2: Average them out and treat them as a single data point? (A user stays in the same group on all days)
In Method 1, my data (illustrative) would like this:
And my sample size for V1 would be 2 and for V2 it would be 1
In Method 2, my data would like this:
And my sample size for V1(Test) would be 1 and for V2(Control) also it would be 1.
I'm trying to identify if variation one causes a user to spend more time on the website per day.
(P.S. the data is illustrative. I will have thousands of users)