0
$\begingroup$
  1. How do I choose the right sample size for A/B test? looks like if I increase the sample size the test can always result in positive

  2. Do we use t-test if my sample size is more than 30? because in t-test we use the standard error(SE) and the standard error will become small when the sample size is huge. if SE is less then my t-test will show positive.

looks like I am missing something, please help

$\endgroup$
5
  • 1
    $\begingroup$ To estimate the required sample size with depend on what you are measuring. Is it ratio or a measured value? The sample will also depend on how small of a difference you are looking to detect with a given confidence limit. If the difference is large between A vs B then a smaller sample size can be used to. If the difference is small, then a large number of samples is needed to separate the signal from the noise. $\endgroup$
    – Dave2e
    Commented Apr 23, 2020 at 11:49
  • $\begingroup$ it is a measured value. In general, my understanding is more the sample size better my test result would be but, why is it not the case here? $\endgroup$
    – Angadishop
    Commented Apr 23, 2020 at 12:56
  • 1
    $\begingroup$ I agree with you that larger the sample size the better. I am not sure what you meant by " increase the sample size the test can always result in positive". Also note that as sample size increases the t-test becomes more normal distribution. Thus for greater than 30 samples the t-test and z-test becomes same test. $\endgroup$
    – Dave2e
    Commented Apr 23, 2020 at 14:26
  • $\begingroup$ @Dave2e How do we A/B test for a ratio? can you please point to relevant material. $\endgroup$
    – Angadishop
    Commented May 23, 2020 at 20:32
  • $\begingroup$ There are a couple of options depending on your data. The basic is Z Score for 2 Population Proportions. Another option is the Fisher Exact test. With the Fisher's test You would have a 2x2 contingency table with the number of Yes and No for both your A and B conditions. $\endgroup$
    – Dave2e
    Commented May 24, 2020 at 3:29

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.