Let's say I have $n$ pairs of items. For each pair I select one item over another due to some perceived trait. This could be anything, for example, identifying which image is brighter than the other or something.

I want to then do this exact same test with a bunch of participants and determine if their perception agrees with mine.

My initial thought was to collect the data, and then do a set of binomial tests for each pair, the null hypothesis being that they make the same choice as me, thus treating my choice as an expected value (p=1 or p=0 of picking item 1 over item 0 for example). Does this make sense? Or is there some way I can use the chi-squared test or something? I'm a novice when it comes to these sorts of statistical tests.

  • $\begingroup$ There might be interested in cohen's kappa $\endgroup$ – Demetri Pananos Aug 14 '19 at 17:49
  • $\begingroup$ Does 'same as mine' mean making the same individual choices you did, or having the same number of 'successes' as you did. (Where 'success' means 'rated as bright'.) $\endgroup$ – BruceET Aug 15 '19 at 21:31
  • $\begingroup$ @BruceET Same individual choices, because for each pair I make one choice only. $\endgroup$ – Jonathan Aug 15 '19 at 23:19
  • 1
    $\begingroup$ Then here's one idea: Consider scoring each of $m$ others on their nr of matches of your choices: scores 0, 1, 2, ..., 10. By chance their scores should be BINOM(n, .5). Use $m$ scores to test $H_0: p=.5$ vs $> .5.$ Rejection means they tend to agree with you. 95% CI for $p$ will show extent to which they agree with you. // If you're only interested in exact agreement with you, this wouldn't be the approach. But why would you expect people to agree with you every time? $\endgroup$ – BruceET Aug 16 '19 at 1:15
  • $\begingroup$ @BruceET Ah so a binomial test for binomial distribution of correct choices in total for all pairs? That's a good answer, can't believe I didn't think of that. For some weird reason I was thinking I'd have to do an individual binomial test for each pair. This makes much more sense. $\endgroup$ – Jonathan Aug 16 '19 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.