I am performing A/B testing (basically hypothesis testing) with relatively small samples, so the results are largely inconclusive. I am aware of techniques like CUPED (for decreasing the sample variance and hence increasing the experiment sensitivity) or the objective Bayes approach (although I'm not entirely sure how it would perform with small samples), or using Wilcoxon test rather than a t-test, but any other suggestion will be greatly appreciated.
1
-
4$\begingroup$ If you've got a small sample, you don't have much information. You can't get more information, without a bigger sample. $\endgroup$– Jeremy MilesCommented Jan 31, 2021 at 18:30
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I would say that the business conclusion is that you have too noisy of a process to assess using current resources. If you want to get a stronger conclusion, you will need to invest more in acquiring data. You can use power and sample size calculations to guide you to the needed data size and how expensive it would be to obtain such data (if it’s €10,000 per observation and you need 50 new observations...).
It is an acceptable business decision to decide that company money is scarce and it is not worth spending €500,000 on this project. You then can play games, through power and sample size calculations, to determine what you can say if €500,000 is unacceptable but €200,000 (for instance) is. Perhaps you have to settle for being able to detect larger differences, but perhaps that still can be valuable.
$\begingroup$
$\endgroup$
Bayes works on small samples, it's just that it's highly sensitive to what priors you use; unless your data strongly points towards one conclusion, your posterior won't differ hugely from your posterior. Depending on what you're using this for, it may be best to calculate an odds ratio and hope that your audience knows how to interpret that correctly. If you have to choose between A or B, and the only information you have is that one did better than the other on a small sample test, then you're probably best off going with that one, although there is a danger that "This is our best guess" will evolve into people believing "We've verified that this is the best one".
