The probability for two people to provide identical answers on survey questions

Here is the problem: A survey contains 7 binary questions (Yes/No responses). If two people are answering the survey, what is the probability for their answers on 4 or more of the questions to match? In other words, if we have four or more matching answers, we can consider the overall survey response to be similar for both people.

• added r tag, so the R code in the comment is highlighted – mpiktas Jan 10 '11 at 7:17
• @mpiktas This question is quite independent of R, IMHO. Moreover, code highlighting is handled through Google Prettify on SE sites, so there's no need to add extra tag to enable syntax highlight. – chl Jan 10 '11 at 9:10
• @chl, I asked the question on meta concerning adding r tags, since I felt too, that adding r tag just for syntax highlighting is not appropriate. Google prettify did not work for me. – mpiktas Jan 10 '11 at 9:22

I assume that the survey will be answered independently by the participants. First, you need estimates for the baseline probabilities $p_{i}$ that an answer $i$ will be answered "yes". The probability of two persons answering "yes" for question $i$ is then $p_{i}^{2}$. Likewise, the probability of two persons answering "no" for question $i$ is $(1-p_{i})^{2}$, hence the probability of agreement is $p_{i}^{2} + (1-p_{i})^{2}$.

If you assume that all $p_{i} = 0.5$, then you get the answer given by carlosdc since $0.5^{2} + (1-0.5)^{2} = 0.5$. If you allow the $p_{i}$ to vary, an answer can probably be given in closed form as well, but with only 7 questions, it's easy to simply enumerate all possibilities to get 4 or more agreements, and calculate the probability for each case.

> n <- 7            # number of questions
> p <- rep(0.5, n)  # probabilities p_i, here: set all to 0.5
# p <- c(0.4, 0.4, 0.4, 0.4, 0.1, 0.1, 0.1) # alternative: let p_i vary
> k <- 4:7          # number of agreements to check
# k <- 0:7          # check: result (total probability) should be 1

# vector to hold probability for each number of agreements
> res <- numeric(length(k))

# function to calculate the probability for an event with agreement on the
# questions x and disagreement on the remaining questions
> getP <- function(x) {
+     tf <- 1:n %in% x              # convert numerical to logical index vector
+     pp <- p[tf]^2 + (1-p[tf])^2   # probabilities of agreeing on questions x
+
+     # probabilities of disagreeing on remaining questions
+     qq <- 1 - (p[!tf]^2 + (1-p[!tf])^2)
+     prod(pp) * prod(qq)           # total probability
+ }

# for each number of agreements: calculate probability
> for(i in seq(along=res)) {
+     # all choose(n, k) possibilities to have k agreements
+     poss <- combn(1:n, k[i])
+
+     # probability for each of those possibilities, edit: take 0-length into account
+     if (length(poss) > 0) {
+         res[i] <- sum(apply(poss, 2, getP))
+     } else {
+         res[i] <- getP(numeric(0))
+     }
+ }

> res                # probability for 4, 5, 6, 7 agreements
[1] 0.2734375 0.1640625 0.0546875 0.0078125

> dbinom(k, n, 0.5)  # check: all p_i = 0.5 -> binomial distribution
[1] 0.2734375 0.1640625 0.0546875 0.0078125

> sum(res)           # probability for 4 or more agreements
[1] 0.5


The R code could certainly be simplified, also prod() might be worse in terms of error propagation with small numbers than exp(sum(log())), although I'm not sure on that one.

• The "closed formula", expressed as a polynomial in the original $p_i$, has 2187 terms--more than the number of values you would have to enumerate! – whuber Jan 10 '11 at 2:37
• Yes and No answers have a probability of 0.5. This is correct: 0.2734375 0.1640625 0.0546875 0.0078125. There are 16 possible combination of Yes answers for two users: {4,4} {4,5} {4,6} {4,7} {5,4} {5,5} {5,6} {5,7} {6,4} {6,5} {6,6} {6,7} {7,4} {7,5} {7,6} {7,7}. And we can estimate the probability of each of these 16 pairs occuring. However once a pair occurs - the probability of a match is different. In other words for a {4,4} the probability is 1/35 as there are 35 combinations of 4 out of 7. For {4,5} the probability of a match increases. How do we take those in account. – user2715 Jan 10 '11 at 16:20
• I'm not sure I fully understand your question, but in the for() loop, each number of pairwise agreements is considered separately (4, 5, 6, 7 in your case): for each of these numbers, all possible answer patterns leading to that agreement are enumerated (poss), and their respective probabilities calculated. One way to check that the result is correct, you can set k (all numbers of agreement) to 0:7. The sum for all probabilities should then be 1. – caracal Jan 10 '11 at 19:11
• @Rado It looks like you are conditioning on the numbers of yes answers; e.g., {6,4} means respondent 1 had 6 yeses and respondent 4 had 4 yeses. This is a valid approach but is more complex than necessary. @caracal is modeling it question by question: on question $i$, the chance of agreement is like flipping a coin with probability $p_i^2 + (1-p_i)^2$. By looking at all possible outcomes of these 7 coin flips--there are only 128 of them--we can compute the chance of any pattern of agreement. – whuber Jan 10 '11 at 20:37
• I might be missing somenthig - ( res ) is probability of selecting 4, 5,6 or 7 "yes", e.g. for 4 it is 35/128. sum (res) - are you suggesting that this is the probability of the the two picking the same answers? To me sum (res) is the probability of anyone selecting 4 or more Yes out of all possible combinations. – user2715 Jan 10 '11 at 20:40

If for each question the probability of selecting the same answer is equal to 0.5, the answer is the following:

$$\sum_{i=4}^7{\binom{7}{i}p^i(1-p)^{7-i}}$$

where $p=0.5.$

In this case it is a binomial distribution.

• It's simpler to note that the probability of 4 or more agreements equals the probability of 3 or fewer and, since that exhausts all possibilities, both numbers must equal 1/2 exactly. – whuber Jan 10 '11 at 2:30
• You're right. I just wanted to give an answer based on undergraduate probability and give where it comes from. Having the formula also gives you insight as to how the value changes for different i's and p's. – carlosdc Jan 14 '11 at 5:28