0
$\begingroup$

Let's say we conduct an A/A test for testing app.

We will randomly divide users into two groups, with a 50% probability of each user getting into one or the other of the groups, but both groups will get the exact same app.

The onboarding conversion rate for these groups turned out to be different, lets say because of seasonality.

My question is if we add more users to the previous A/A test, will the difference in conversions calculated based on the users in each of the groups increase or decrease?

Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

There are a few things going on in this that have to be considered. First, the overall question: Either could happen. The final difference could get larger or smaller. It depends on the heterogeneity of the population and how representative each sampling was/would be, as well as what the underlying reality you're trying to measure turned out to be.

Furthermore, since you will have sampled twice, you've added another level to the model, which may need to be accounted for.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you. I got the following official answer: That's wrong. Imagine that you have two identical coins. You start tossing each of them and count the percentage of heads against tails. The more you toss the coins, the closer the value will be to 50% for each of the coins, and thus, the closer they’ll match up to each other. Part 1 $\endgroup$
    – Ivana Bbb
    Commented Feb 22, 2023 at 21:15
  • $\begingroup$ Part 2 : We have a completely identical situation with onboarding conversion using the A/A test. The more users there are in an experiment, the closer the calculated conversion rate estimates will be to real onboarding conversion rates, which is exactly the same for the two versions. Therefore, the conversion rate estimates will tend to match each other with more users added to the experiment. $\endgroup$
    – Ivana Bbb
    Commented Feb 22, 2023 at 21:16
  • $\begingroup$ I believe their answer does not stand huh? $\endgroup$
    – Ivana Bbb
    Commented Feb 22, 2023 at 21:16
  • $\begingroup$ This "official answer" sounds correct. If A/A test mean you're sampling from the same distribution and comparing the difference, by the law of large numbers, the difference converges to zero as n goes to infinity. Of course, for a small number of additional samples, you could get a difference that's bigger than your previous difference (as Byran notes above) but on average, it will be smaller. $\endgroup$
    – num_39
    Commented Feb 22, 2023 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.