2
$\begingroup$

I know similar questions have been asked many times here. But I still don't get simple formula or idea how estimate this.

I have site with traffic and I want to split portion of it to test new search relevance algorithms, i.e., result sorting. I will measure abandonment rate (searches without click/total amount of searches). I want to get this statistics with $\pm2\%$ accuracy with $95\%$ confidence.

Is there simple heuristic (something practical) I can apply to get required number of experiments ($10^3,~10^6,~\ldots$)?

$\endgroup$
2
$\begingroup$

Your question relates to the idea of statistical power. There are a lot of threads on CV that provide information to help you think about this, search under: and . In particular, you may want to read through this answer of mine: How to report general precision in estimating correlations within a context of justifying sample size, which discusses the idea of Accuracy in Parameter Estimation--a framework similar to power, but specific to the question of how much data you need to get a confidence interval down to a certain size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.