I have paired data from 9 individuals. The response variable is ordinal with levels 1-5. As the sample is very small and the data are categorical, I thought the appropriate test was McNemar's test. However I get McNemar's chi-squared = NaN and therefore a p-value = NA. What should I do to test whether the effect is significant?

My data:


4 3 4 5 4 4 4 5 3


5 4 3 5 5 4 5 5 4

I wonder if you really want McNemar's test (more specifically the McNemar-Bowker test). These are tests to see if the marginal proportions are the same (see here). Under the assumption that pairings can be recovered from the orders of the two vectors, here is a table of your data with the marginal proportions computed:

bef = c(4, 3, 4, 5, 4, 4, 4, 5, 3)
aft = c(5, 4, 3, 5, 5, 4, 5, 5, 4)
table(bef, aft)
#    aft
# bef 3 4 5
#   3 0 2 0
#   4 1 1 3
#   5 0 0 2
# bef
#         3         4         5 
# 0.2222222 0.5555556 0.2222222 
# aft
#         3         4         5 
# 0.1111111 0.3333333 0.5555556 

Since you state that your variables are ordered, it seems more appropriate to assess if one set has higher values than the other. That would use the Wilcoxon signed rank test:

wilcox.test(bef, aft, paired=TRUE)
#   Wilcoxon signed rank test with continuity correction
# data:  bef and aft
# V = 3.5, p-value = 0.1294
# alternative hypothesis: true location shift is not equal to 0

Because you have ties, you cannot use the exact version of the test; you use the asymptotic approximation. If that bothers you due to the small sample size, you could simulate the null and compute the p-value that way. There are different ways to do that, but I'm not sure how much difference they will make in practice. Bootstrapping is generally not recommended with small samples. You could try a permutation-based version, or you could do a Monte-Carlo simulation of the null based on the marginal distribution of the values (ignoring timepoint).

  • $\begingroup$ Thank you very much, I thought the Wilcoxon signed rank test would not be valid for qualitative data, because it doesn't take into account that the response variable can only take values from 1 to 5. $\endgroup$ – Mik meadow Jul 26 '16 at 20:11
  • 1
    $\begingroup$ The only assumption the Wilcoxon signed rank test makes about your response data is that you can say if 1 value is > another. You can do that here, so there isn't really a problem. You cannot use the exact version of the test since there are ties; you use the asymptotic approximation. If that bothers you due to the small sample size, you could simulate the null and compute the p-value that way. $\endgroup$ – gung - Reinstate Monica Jul 26 '16 at 20:19

It is almost certainly because you have a pair of cells which both have zeroes (3, 5) and (5, 3). If you look at the way the test is defined that will lead to problems.

What to do is a problem. You could always merge categories but that may not make scientific sense. I think you may have to program your own solution. You could define a log-linear model of quasi-symmetry without including the offending cells (treating them as structural zeroes) but that may not be a sensible assumption either. More data would help but I suppose that may not be an option.

  • $\begingroup$ Thanks, so how can I test it in R? $\endgroup$ – Mik meadow Jul 26 '16 at 17:09
  • $\begingroup$ @Mikmeadow I have edited my answer $\endgroup$ – mdewey Jul 26 '16 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.