# Which significance test is the best for measuring the response to two survey questions are significantly different?

I have a survey where there are two questions designed in a Likert Scale (1-7). Let's say:

By counting the frequency of the likert response. I found out more people rated apples higher. Now, I want to know if people rate apples significantly differently than pears.

So, it's the same group of people, and no time difference (like pre/post). And the response are ranked. What's the most appropriate tool for this situation?

• If you are willing to treat (ordinal) Likert-7 data as if it were numerical, you might be able to use a paired t test, provided Likert scores are approximately normal (roughly symmetrical with no extreme outliers). Otherwise, perhaps, a paired Wilcoxon signed rank test (unless excessive ties are a problem)). // One may wonder why there was no straightforward question directly comparing apples and pears. (0 = Very strongly prefer pears, ..., 7 = Very strongly prefer applies.) May 12 at 23:00

It would be easier to answer directly if you had provided data. I will answer by illustrating with fictitious data. If issues arise with your data that are not covered here, please edit your question to provide pertinent information, and maybe someone will respond.

Suppose the sample size is $$n = 200$$ subjects, and that the sample distribution of the differences d between Likert scores for Apples and Pears is as follows.

mean(appl); mean(pear)
[1] 4.815
[1] 4.545
cor(appl, pear, meth="spearman")
[1] 0.783856


On average, there are higher scores for Apples, and people who like one seem to like the other (Spearman rank correlation is $$0.78.)$$

d = appl - pear
table(d)
d
-2 -1  0  1  2
15 29 70 59 27


A paired Wilcoxon signed rank test is a test on the differences 'd'. There are many tied values, but with $$n$$ as large as 200, the P-value is an approximation that deals appropriately with ties. The null hypothesis that the population is centered at $$0$$ is strongly rejected with P-value about $$0.001 < 0.05 = 5\%.$$

wilcox.test(d)

Wilcoxon signed rank test
with continuity correction
data:  d
V = 5582, p-value = 0.001299
alternative hypothesis: true location is not equal to 0


Even with $$n=200$$ it seems quite a reach to view a discrete distribution with only five integer values as normal, but for the record a paired t test also gives a very small P-value.

t.test(d)\$p.val
[1] 0.0006454586


Finally, of the $$200 - 70 =130$$ subjects who expressed a clear preference, only $$14+29=44$$ preferred pears. So a 2-sided exact binomial test rejects equal preferences for apples and pears with P-value $$0.0003 < 0.05 = 5\%.$$

pbinom(44, 130, .5)*2
[1] 0.0002895488


Note: Here is how I used R to sample the fictitious data used in the samples above.

set.seed(123)
appl = sample(1:7, 200, rep=T, p=c(1,2,3,4,5,5,4) )
dif = sample(-2:2, 200, rep=T, p=c(1,1,2,2,1))
pear = appl - dif
pear[pear>7]=7;  pear[pear < 1]=1