How to infer on a human intelligence task about selecting the best option I have a human intelligence task test which tries to get the best option from a set of options tested against a control group.
The test consist of:


*

*Suppose we have a control group of options, lets say 10.

*Each option is related to a common characteristic or concept, but they are different from each other and ordered, descending from the most relevant one.

*We focus on one option and call it the control variant.

*We build n variants of this option and call them test variants.

*Each variant is put in place of the control variant to build a test group.

*The test is run asking a human to select the best option from a group which describes our characteristic or concept. The group (including the control group) that is shown to the test taker is selected uniformly at random. We run the test 500 times, ensuring that one test taker can solve the test just once.



I want to measure which among the control variant and the test variants is better to the test takers. For instance, we could select the variant which would be selected more times with respect to its group options, i.e., if a group is shown to the test takers k_i (with i = 1,2,...,n) times (ideally the same value 500/n), select the variant with higher selection rate with respect its own group options in the k_i samples.
I'm not an expert, but I clearly see that the way I run the test introduces some problems. For example, what clearly establish that this approach would lead to the best variant according to the test takers criteria? How much confident am I on this? Clearly the shown groups won't be perfectly selected uniformly. Will the samples amount lead to a significant result?
Some friend told me that I would be interested on reading about chi-square tests, and sent me this article: https://www.lunametrics.com/blog/2014/07/01/statistical-significance-test, but I don't completely understand if this would be applied to the problem.
How can I infer what is the best option according to the described test with high confidence? Can you share with me some concepts, articles or books to learn about this kind of problems?
 A: You mentioned the following (in a comment):

The "best" option should be the one with more rate of selection among all test takers.

This suggests your statistic of interest is simply the rate of selection $p$ of some option $j$ $(j=1,...,10)$ for a particular group $i$ $(i=1,...,m)$.
Therefore, you can compare the value of $p$ with respect to some $j$ amongst each $i$.  
Because you suggest there will be multiple variants, tests like a two-proportion test won't cut it.  Instead, your main option is (as your friend suggested) a Chi-squared test.

Additional Notes:  


*

*the example you linked for Chi-squared tests mentions using a Bonferroni correction, which would be applicable in your case if you choose to follow this method


*

*however, this correction can often be conservative for a large number $m$ of groups (in which case, you should avoid it)


*the example I linked for Chi-squared tests may be better to follow, since its method generalizes the number $m$ of groups (rather than assuming that there are only two)


*

*in this case, your cross-tab would have $m$ rows (for each control and variant group) and 2 columns (e.g. "selected option j" vs. "did not select option j")




The power of your test, i.e. the likelihood of detecting some effect assuming the effect actually exists (an effect being something along the lines of "option $j$ has a higher selection rate $p$ for some particular group $i$"), is going to heavily rely on your sample size ($\approx \frac{500}{m}$) for each group and the actual magnitude of the effects at hand.  
Without having a good idea of the number of groups and the context of the problem, it is difficult to infer the power of your proposed test design.
The main takeaway is that if the expected difference in selection rates between groups is higher, you could expect your test's statistical power to also be higher.
