How to compare conversion rates in an AB test? Please help me decide how to proceed in the following situation:


*

*Suppose we have an AB test with 2 groups. 

*The target metric is the conversion from one web page to another. 

*Suppose also, that whenever a user enters the website, we are able to identify his ID correctly. 

*Every user can go through the "flow" arbitrary number of times (i.e. he can do the target action 3 out of 7 times).


So here is my question:
Should I:


*

*Calculate the overall conversion rate for both groups and compare it via a simple z-test?

*Calculate the overall conversion rate for each user and then compare the average values across groups via a t-test?
Intuitively, users with a larger number of actions are given more weight in the first option, while in the latter everyone carries equal weight.
Finally, which option do I choose?
 A: I am going to cook up an admittedly  extreme answer to demonstrate my point.  Let's say you run an AB test on your platform to examine conversion.  Whatever you are measuring can be accessed several times, and so you see the same customer pop up in your data more than once.  
We randomize 10 users to versions A or B, and then perform a hypothesis test.  Here is some data I simulated
# A tibble: 2 x 3
  group     z     n
  <chr> <int> <int>
1 A        23    50
2 B       159   230

Here, the sum of the $n$ column is larger than 10 because I see experimental units more than once (as in your first approach).  If I did a test of proportions on this data, then I would reject the null with a p value of about 0.003.  Wow, there is less than a 3 in 1000 chance we get this result assuming there is no difference.  We should ship this change!
Or should we?  Let's see how many times each subject was observed.
  users     n
   <fct> <int>
 1 1        10
 2 2        10
 3 3        10
 4 4        10
 5 5        10
 6 6        10
 7 7        10
 8 8        10
 9 9       100
10 10      100

We saw 8 of the 10 experimental units a total of 10 times, but we saw the last two units 100 times!  If these users were more likely to convert anyway regardless of experimental arm, then our results are biased!  And that is exactly what happened.
# A tibble: 10 x 4
   group users     z     n
   <chr> <fct> <int> <int>
 1 A     1         4    10
 2 A     2         4    10
 3 A     3         5    10
 4 A     4         4    10
 5 A     5         6    10
 6 B     6         7    10
 7 B     7         6    10
 8 B     8         3    10
 9 B     9        74   100
10 B     10       69   100

These users were just more likely to convert even before the experiment.  This happens in real life.  If you buy a lot of stuff on Amazon, for example, you probably are not going to be affected by experimental arm, which can lead to stuff like this happening.
How can we get around this?  Previously, the more we saw an experimental unit, the more weight they had.  To weight everyone accordingly we can estimate each person's probability of conversion and then do a t test.  When we do that, we correctly fail to reject the null (when I generated this data, I made sure there was no difference between groups but that experimental units had their own unique probability of conversion).
