I have a human intelligence task test which tries to get the best option from a set of options tested against a control group.

The test consist of:

  • Suppose we have a control group of options, lets say 10.
  • Each option is related to a common characteristic or concept, but they are different from each other and ordered, descending from the most relevant one.
  • We focus on one option and call it the control variant.
  • We build n variants of this option and call them test variants.
  • Each variant is put in place of the control variant to build a test group.
  • The test is run asking a human to select the best option from a group which describes our characteristic or concept. The group (including the control group) that is shown to the test taker is selected uniformly at random. We run the test 500 times, ensuring that one test taker can solve the test just once.

enter image description here

I want to measure which among the control variant and the test variants is better to the test takers. For instance, we could select the variant which would be selected more times with respect to its group options, i.e., if a group is shown to the test takers k_i (with i = 1,2,...,n) times (ideally the same value 500/n), select the variant with higher selection rate with respect its own group options in the k_i samples.

I'm not an expert, but I clearly see that the way I run the test introduces some problems. For example, what clearly establish that this approach would lead to the best variant according to the test takers criteria? How much confident am I on this? Clearly the shown groups won't be perfectly selected uniformly. Will the samples amount lead to a significant result?

Some friend told me that I would be interested on reading about chi-square tests, and sent me this article: https://www.lunametrics.com/blog/2014/07/01/statistical-significance-test, but I don't completely understand if this would be applied to the problem.

How can I infer what is the best option according to the described test with high confidence? Can you share with me some concepts, articles or books to learn about this kind of problems?

  • 1
    $\begingroup$ This seems to be mostly about experiment design. I am confused by the names that are used and the test design. But the goal of the research is almost clearly defined: which choice is better to test takers. You'd need a solid definition of what "better" means, at best numerically. Then you might be able to determine a confidence level for this number. As I said, I'm confused about what you want to determine: the best experiment design or the result of the experiment. Are all groups shown to a test taker or just a single group? $\endgroup$ – cherub Jun 14 '18 at 12:26
  • $\begingroup$ @cherub Yes, you are right, "better" is a concept that should be defined. It's an experiment in which just a single group is shown to a test taker. The shown group is selected uniformly from all the groups (including the control group). A test taker receives a group and then selects an option among the ten options in the group. I'm not interested in measuring which of the ten options is selected more times, but what options among test variants and control variant is selected the most. The "best" option should be the one with more rate of selection among all test takers. $\endgroup$ – OSjerick Jun 14 '18 at 17:39

You mentioned the following (in a comment):

The "best" option should be the one with more rate of selection among all test takers.

This suggests your statistic of interest is simply the rate of selection $p$ of some option $j$ $(j=1,...,10)$ for a particular group $i$ $(i=1,...,m)$.

Therefore, you can compare the value of $p$ with respect to some $j$ amongst each $i$.

Because you suggest there will be multiple variants, tests like a two-proportion test won't cut it. Instead, your main option is (as your friend suggested) a Chi-squared test.

Additional Notes:

  • the example you linked for Chi-squared tests mentions using a Bonferroni correction, which would be applicable in your case if you choose to follow this method
    • however, this correction can often be conservative for a large number $m$ of groups (in which case, you should avoid it)
  • the example I linked for Chi-squared tests may be better to follow, since its method generalizes the number $m$ of groups (rather than assuming that there are only two)
    • in this case, your cross-tab would have $m$ rows (for each control and variant group) and 2 columns (e.g. "selected option j" vs. "did not select option j")

The power of your test, i.e. the likelihood of detecting some effect assuming the effect actually exists (an effect being something along the lines of "option $j$ has a higher selection rate $p$ for some particular group $i$"), is going to heavily rely on your sample size ($\approx \frac{500}{m}$) for each group and the actual magnitude of the effects at hand.

Without having a good idea of the number of groups and the context of the problem, it is difficult to infer the power of your proposed test design.

The main takeaway is that if the expected difference in selection rates between groups is higher, you could expect your test's statistical power to also be higher.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.