# How do I determine goodness of fit for a test of n binary decisions?

Let's say I have $$n$$ pairs of items. For each pair I select one item over another due to some perceived trait. This could be anything, for example, identifying which image is brighter than the other or something.

I want to then do this exact same test with a bunch of participants and determine if their perception agrees with mine.

My initial thought was to collect the data, and then do a set of binomial tests for each pair, the null hypothesis being that they make the same choice as me, thus treating my choice as an expected value (p=1 or p=0 of picking item 1 over item 0 for example). Does this make sense? Or is there some way I can use the chi-squared test or something? I'm a novice when it comes to these sorts of statistical tests.

• There might be interested in cohen's kappa – Demetri Pananos Aug 14 '19 at 17:49
• Does 'same as mine' mean making the same individual choices you did, or having the same number of 'successes' as you did. (Where 'success' means 'rated as bright'.) – BruceET Aug 15 '19 at 21:31
• @BruceET Same individual choices, because for each pair I make one choice only. – Jonathan Aug 15 '19 at 23:19
• Then here's one idea: Consider scoring each of $m$ others on their nr of matches of your choices: scores 0, 1, 2, ..., 10. By chance their scores should be BINOM(n, .5). Use $m$ scores to test $H_0: p=.5$ vs $> .5.$ Rejection means they tend to agree with you. 95% CI for $p$ will show extent to which they agree with you. // If you're only interested in exact agreement with you, this wouldn't be the approach. But why would you expect people to agree with you every time? – BruceET Aug 16 '19 at 1:15
• @BruceET Ah so a binomial test for binomial distribution of correct choices in total for all pairs? That's a good answer, can't believe I didn't think of that. For some weird reason I was thinking I'd have to do an individual binomial test for each pair. This makes much more sense. – Jonathan Aug 16 '19 at 9:51