# What criterion to use to compare multiple correlations of binary variables?

I have $$N$$ definitions of certain properties of countries (for example, if the country is "democratic", "totalitarian" etc.), and want to test how consistently different people understand these definitions. For that, I create a questionnaire containing for each definition $$k$$ yes/no questions of the form "Does the following country satisfy the definition above?", followed by a "de-personified" description of a country. My null hypothesis is that all definitions are equally consistent, and the alternative hypothesis is that some of the definitions is more or less consistent than the rest. By consistency I mean "how much classifications performed by different people correlate one with another".

My questions are: 1) What correlation metric should I use, and 2) What statistical criterion should I employ to test homogeneity of correlations?

UPD: I've realized there was a mistake in my original description of the problem (besides, it was not clear enough), and have rewritten the question from scratch.

• This sounds like interrater agreement with binary ratings. Commented Jul 15 at 3:10
• Yes, thank you for the name, looks like the thing I need Commented Jul 15 at 8:24

You could try formulating this as, all students are exchangeable, so probability of $$i-th$$ question to be answered as yes is $$\pi_i$$, i.e. each answer to i-th questions from m-th student is $$A^{(m)}_i\sim Bernoulli(\pi_i)$$. In which case the number of yes-answers from $$M$$ students will be distributed as $$Y_i\sim Binomial(M,\,\pi_i)$$. The probability distribution for $$\pi_i$$ will then be given by Beta distribution. If you want to stick to frequentist methods your null hypothesis would be $$Y_i\sim Binomial\left(M,0.5\right)$$ (if I understood your definition of 'not formulated clearly'). For a good alternative hypothesis you will need to specify just how far away from $$\pi_i=0.5$$ a clearly formulation has to be (as a minimum).
• Thanks for the attempt! Unfortunately, your idea does not help me for two reasons. The first (and a minor one) is that initially I did not formulate the problem the way I meant it. I've since rewritten the description from scratch. I hope, the new problem statement is more clear, too. The second is that, since I don't know myself if the definitions are consistent, I cannot presume a specific proportion of "yes" and "no" answers. Therefore, there is no reason to expect that, given $H_{0}$, the binomial distribution should have a specific value of its parameter $p$. Commented Jul 15 at 8:19