# Sample size calculation for precision at $k$: a/b-testing a recommender system

I would like to conduct an online experiment to compare two different versions of a recommender system. The system returns a list of $$r$$ ranked recommendations. I would like to evaluate the recommender versions using the precision at $$k$$, where $$k \leq r$$, and $$k$$ is a small number, usually with $$k \leq 10$$. The precision at $$k$$ will be calculated as the share of the first $$k$$ recommendations which receive a click within the same web session. So if a users clicks on 2 of the first 5 items, the precision at 5 is 2/5 = 0.4. The individual unit of experimentation is the web session (and the recommendations don't change within a session), and we will average the precision at $$k$$ over all web sessions within each group to compare the two versions of the recommender.

My question is: How large a sample do I need?

For $$k=1$$, this could be calculated with standard formulas for comparing binomial proportions, because the item on the first position is either being clicked on or not, and the metric we look at equals the number of clicks, divided by the number of sessions. Of course, that needs the usual assumptions on parameters like significance level and sample size.

But how can the sample size be estimated for $$k>1$$?