1
$\begingroup$

I have 5 point Likert scale answers to questions and I want to compare the results for 2 practice types. I have only 17 respondents from one type so I believe my best approach is the Fisher exact test.

The articles I can find that do something like this seem to just look at the responses for agree & strongly agree. I'm assuming they must be running the test with # A/SA responses and the number of all other responses (neutral, disagree and strongly disagree). So if there are 2 categories, they have a 2x2 matrix of data they are testing for each question.

On the one hand it seems like you'd want to you all of the data you have an not condense it down so much. On the other hand, by doing that you're making it easier to reach significance because you're distinguishing between all of the Likert values, where you weren't before. And this test doesn't understand that the data is ordinal, which means you're losing information.

Are these simply limitations of what I can do with this data? Should I use the 2x2 matrix of data or the 5x2 data (all of the detail)?

Thank you!

$\endgroup$
4
  • $\begingroup$ Any reason, not to use a Mann-Whitney U or Wilcoxon test? $\endgroup$
    – Bernhard
    May 14, 2018 at 14:38
  • $\begingroup$ Welcome to CV! On this site there's no need to say "thank you" at the end of your post - it might seem rude at first, but it's part of the philosophy of this site (tour) to "Ask questions, get answers, no distractions" and it means future readers of your question don't need to read through the pleasantries. $\endgroup$ May 14, 2018 at 15:11
  • $\begingroup$ Regarding those other tests, don't they require that the distrib and variance of the 2 be the same? For some of my answers, they are not, just by inspection. The answers are clustered more heavily at the end of the Likert scale for one practice type than the other. $\endgroup$
    – user208267
    May 14, 2018 at 19:35
  • $\begingroup$ [My comments got long, so I converted to answer.] $\endgroup$ May 15, 2018 at 14:57

1 Answer 1

1
$\begingroup$

It is usually better to treat Likert responses as ordinal data rather than nominal data, and it is usually undesirable to dichotomize data as you are suggesting. Why would you want to throw away the difference between someone answering "SA" rather than "A", or especially "SD" rather than "N"? It is usually best to keep all five categories and treat them as ordinal data, choosing a test that knows that SA > S > N, not just that those are different categories.

As @Bernhard suggests, a Wilcoxon-Mann-Whitney test is commonly used in this case. You will find some disagreement about the appropriateness of this test in this situation, but it performs reasonably well when compared with ordinal regression (See plots at the bottom of this page). There is also the Cochran-Armitage test, and ordinal regression. In any case, be sure to understand the hypotheses and assumptions of whatever test you choose.

As to your question in the comment. (As far as I understand; non-statistician answer.) No, W-M-W does not assume that the distribution and variance be the same among the groups. Unfortunately, there is a lot of confusion about this in the literature --- including journal articles. If some of these assumptions are made, the test can be used as a test of medians. But the test is originally devised as a test of stochastic equality, which I think is more interesting usually anyway. If you wanted to test medians, you could use Mood's median test.

$\endgroup$
5
  • $\begingroup$ What about the issue of ties? I've read that M-W U test is not appropriate when there are lots of ties. With 5 Likert values and 200+ observations (mostly in the traditional delivery model), i will have a very large number of ties. Does that mean I can't use the test? $\endgroup$
    – user208267
    May 16, 2018 at 13:31
  • $\begingroup$ Well, that's a claim I've heard, but I don't know if I should buy it. If you look at the plots mentioned above that compare W-M-W to ordinal regression, with Likert item data, it seems to perform well, at least under the conditions of those simulations.... If you are unsure, you can use ordinal regression. $\endgroup$ May 17, 2018 at 21:55
  • $\begingroup$ I think a paper by Conover (1973) addresses using these tests with discrete data, but I haven't read it yet. Mentioned in Conover (1999). $\endgroup$ May 23, 2018 at 18:53
  • $\begingroup$ I'm going to shoot for using MWU, unless the reviewers don't like it. $\endgroup$
    – user208267
    May 24, 2018 at 21:36
  • $\begingroup$ I think MWU is pretty typical in these cases. I doubt there will kickback from reviewers, but in my experience it is madness to try to predict what reviewers will care about. $\endgroup$ May 24, 2018 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.