I have 5 point Likert scale answers to questions and I want to compare the results for 2 practice types. I have only 17 respondents from one type so I believe my best approach is the Fisher exact test.

The articles I can find that do something like this seem to just look at the responses for agree & strongly agree. I'm assuming they must be running the test with # A/SA responses and the number of all other responses (neutral, disagree and strongly disagree). So if there are 2 categories, they have a 2x2 matrix of data they are testing for each question.

On the one hand it seems like you'd want to you all of the data you have an not condense it down so much. On the other hand, by doing that you're making it easier to reach significance because you're distinguishing between all of the Likert values, where you weren't before. And this test doesn't understand that the data is ordinal, which means you're losing information.

Are these simply limitations of what I can do with this data? Should I use the 2x2 matrix of data or the 5x2 data (all of the detail)?

Thank you!

  • $\begingroup$ Any reason, not to use a Mann-Whitney U or Wilcoxon test? $\endgroup$ – Bernhard May 14 '18 at 14:38
  • $\begingroup$ Welcome to CV! On this site there's no need to say "thank you" at the end of your post - it might seem rude at first, but it's part of the philosophy of this site (tour) to "Ask questions, get answers, no distractions" and it means future readers of your question don't need to read through the pleasantries. $\endgroup$ – Jan Kukacka May 14 '18 at 15:11
  • $\begingroup$ Regarding those other tests, don't they require that the distrib and variance of the 2 be the same? For some of my answers, they are not, just by inspection. The answers are clustered more heavily at the end of the Likert scale for one practice type than the other. $\endgroup$ – GBrk May 14 '18 at 19:35
  • $\begingroup$ [My comments got long, so I converted to answer.] $\endgroup$ – Sal Mangiafico May 15 '18 at 14:57

It is usually better to treat Likert responses as ordinal data rather than nominal data, and it is usually undesirable to dichotomize data as you are suggesting. Why would you want to throw away the difference between someone answering "SA" rather than "A", or especially "SD" rather than "N"? It is usually best to keep all five categories and treat them as ordinal data, choosing a test that knows that SA > S > N, not just that those are different categories.

As @Bernhard suggests, a Wilcoxon-Mann-Whitney test is commonly used in this case. You will find some disagreement about the appropriateness of this test in this situation, but it performs reasonably well when compared with ordinal regression (See plots at the bottom of this page). There is also the Cochran-Armitage test, and ordinal regression. In any case, be sure to understand the hypotheses and assumptions of whatever test you choose.

As to your question in the comment. (As far as I understand; non-statistician answer.) No, W-M-W does not assume that the distribution and variance be the same among the groups. Unfortunately, there is a lot of confusion about this in the literature --- including journal articles. If some of these assumptions are made, the test can be used as a test of medians. But the test is originally devised as a test of stochastic equality, which I think is more interesting usually anyway. If you wanted to test medians, you could use Mood's median test.

  • $\begingroup$ What about the issue of ties? I've read that M-W U test is not appropriate when there are lots of ties. With 5 Likert values and 200+ observations (mostly in the traditional delivery model), i will have a very large number of ties. Does that mean I can't use the test? $\endgroup$ – GBrk May 16 '18 at 13:31
  • $\begingroup$ Well, that's a claim I've heard, but I don't know if I should buy it. If you look at the plots mentioned above that compare W-M-W to ordinal regression, with Likert item data, it seems to perform well, at least under the conditions of those simulations.... If you are unsure, you can use ordinal regression. $\endgroup$ – Sal Mangiafico May 17 '18 at 21:55
  • $\begingroup$ I think a paper by Conover (1973) addresses using these tests with discrete data, but I haven't read it yet. Mentioned in Conover (1999). $\endgroup$ – Sal Mangiafico May 23 '18 at 18:53
  • $\begingroup$ I'm going to shoot for using MWU, unless the reviewers don't like it. $\endgroup$ – GBrk May 24 '18 at 21:36
  • $\begingroup$ I think MWU is pretty typical in these cases. I doubt there will kickback from reviewers, but in my experience it is madness to try to predict what reviewers will care about. $\endgroup$ – Sal Mangiafico May 24 '18 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.