# What test is appropriate to analyse the repeated binary outcome survey results?

I have prepared three survey questions and possible options for the participants are binary, that is "yes" or "no".

I repeated the survey before and after an event and the results are presented in percentage. The participants are same for all questions asked before and after the event but number of participants are different. The number of participants for the survey before the event is 50 and after the event is 35 but those 35 are from the set of initial 50 participants. The outcome is as follows:

                 Before                               After
Question 1     10 (10% says yes & 90% says no)      55 (55 % says yes & 45% says no)
Question 2     12                                   14
Question 3     18                                   40


Now I would like to apply the statistics technique and analyze the significance of the event for all the three questions.

Which test will be appropriate for this? Can I use the t-test or ANOVA test which supports the repeated measures.

A Sign Test is a simple solution to this problem (see Sokal and Rohlf, 1995. Biometry, 3rd ed., pp 444).

I have chosen this approach because it avoids distributional assumptions and accounts for the non-independence of observations (observations across time and across questions come from the same participant).

Calculating a new result

We are going to calculate a new result for each of the 35 participants to see if they answered differently after the event.

The new result is based on combining answers from all three questions – we will not look at whether each separate question is answered differently.

If a participant answers with more “yes” answers after the event then record a “+”. If a participant answers with more “yes” answers before the event then record a “-”. If the number of “yes” answers hasn’t changed then record a “0”.

This will look something like this:

If the event had no effect on the answers then we would expect the same number of “+” and “-“. If it did have an effect then either there will be a lot more “+”, or a lot more “-“.

Calculating a Binomial p value

Let’s suppose you have 22 “+” and 11 “-“, with 2 “0”. We ignore the zeroes so we are left with 33 observations in total.

We choose the result with the fewest cases (“-“) and work out the Binomial probability of observing that number of cases or less when the probability is 0.5. This is easy to do in Excel - use the formula BINOM.DIST(11, 33, 0.5, TRUE). For our example this gives a p value of 0.04.

One tail or two tailed test?

The p value calculated can be used directly for a one-tailed test. If you were expecting the event to increase “yes” answers then use this p value. In this example, because 0.04 < 0.05 we conclude the event had an effect on the participants.

If you were not sure whether “yes” answers would increase or decrease after the event, then double the binomial probability and use this as your p value. In this example, because 0.08 > 0.05 we cannot conclude that the event had an effect on the participants.

• why do we need to use cumulative distribution in BINOM.DIST(11, 33, 0.5, TRUE)? Commented Mar 25, 2020 at 14:42
• Because we want the probability of getting 11 "-" or fewer. So, the probability of getting 11 plus the probability of getting 10 plus the probability of getting 9...and so on. Commented Mar 25, 2020 at 14:46
• can't we use the probability mass function because we want for 11 "-" instead of 1 "-" or fewer. Commented Mar 25, 2020 at 15:10
• No. We are interested in the tail of the Binomial distribution. It's exactly the same idea as in a t-test. Let's say we got a t-statistic of -2. The one tailed p-value is the probability of getting less than or equal to the observed t-statistic of -2. Commented Mar 25, 2020 at 16:09
• If this answer was useful, please can you show that to other users by accepting it. Thank you. Commented Mar 27, 2020 at 8:15