0
$\begingroup$

  1. How do I choose the right sample size for A/B test? looks like if I increase the sample size the test can always result in positive

  2. Do we use t-test if my sample size is more than 30? because in t-test we use the standard error(SE) and the standard error will become small when the sample size is huge. if SE is less then my t-test will show positive.

looks like I am missing something, please help

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ To estimate the required sample size with depend on what you are measuring. Is it ratio or a measured value? The sample will also depend on how small of a difference you are looking to detect with a given confidence limit. If the difference is large between A vs B then a smaller sample size can be used to. If the difference is small, then a large number of samples is needed to separate the signal from the noise. $\endgroup$
    – Dave2e
    Apr 23, 2020 at 11:49
  • $\begingroup$ it is a measured value. In general, my understanding is more the sample size better my test result would be but, why is it not the case here? $\endgroup$
    – Angadishop
    Apr 23, 2020 at 12:56
  • 1
    $\begingroup$ I agree with you that larger the sample size the better. I am not sure what you meant by " increase the sample size the test can always result in positive". Also note that as sample size increases the t-test becomes more normal distribution. Thus for greater than 30 samples the t-test and z-test becomes same test. $\endgroup$
    – Dave2e
    Apr 23, 2020 at 14:26
  • $\begingroup$ @Dave2e How do we A/B test for a ratio? can you please point to relevant material. $\endgroup$
    – Angadishop
    May 23, 2020 at 20:32
  • $\begingroup$ There are a couple of options depending on your data. The basic is Z Score for 2 Population Proportions. Another option is the Fisher Exact test. With the Fisher's test You would have a 2x2 contingency table with the number of Yes and No for both your A and B conditions. $\endgroup$
    – Dave2e
    May 24, 2020 at 3:29

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.