I couldn't find this situation described somewhere, though I think it's pretty common so I post it here. I have data from survey that is asking repondents to evaluate their color preferences across different types of webpages. There are 10 colors and 10 webpages, every person gives Likert type answer (5 choices) for each combination.

I would like to do two things:

1) For each website determine whether there is significant difference among color preferences (and ideally group similar colors together).

2) Test differences in count distributions of Likert categories across websites.

I end up using Friedman test for 1), because I consider questions as repeated measures on the same sample. Is this appropriate here? Or is there a better choice, because Friedman test is reported to be of low power?

What about post-hoc after Friedman in my case when there is 45 hypothesis for each website?


EDIT: Edits in response to comments.

Friedman test is used for unreplicated complete block design. In this design, aside from the dependent variable, there is one independent variable, one blocking independent variable, and only one observation per each cell. It sounds like you will be able to use this test on individual webpages.

The power for Friedman test is generally considered okay if there are more than 5 or 6 groups. (For a smaller number of groups, Quade test is sometimes recommended.) One post-hoc test for Friedman is the Conover test. Another approach would be to use pairwise sign tests with a p-value correction for multiple tests.

If you would like to construct a more complex model, perhaps using one model for all webpages together, probably the best approach is ordinal regression. This is as flexible as OLS multiple regression, except that the dependent variable is ordinal in nature, as Likert responses are.

Ordinal regression is pretty easy in R, and is also available in SAS and SPSS. (I don't have any direct experience with ordinal regression in SAS or SPSS.) In R, at least, you can also make it a mixed-effects model, which might be applicable in your case.

  • $\begingroup$ Do you mean that design is not appropriate because of three independent variables? Because my initial goal was to use this for each website separately. Then I don't see any problems to fit data into complete block design. $\endgroup$ – Iden Feb 12 '18 at 7:57
  • $\begingroup$ (Comments deleted and integrated into answer.) $\endgroup$ – Sal Mangiafico Feb 12 '18 at 20:59

I would use a Mann-Whitney U-test to test whether for a particular colour (or a group of similar colours) the responses come from the same population. According to my knowledge, this is the most commonly used statistical test for Likert-type data. I would also compare the median of the statistics, to compare central tendencies of the samples, because this is an ordinal data, so the mean would not make any sense.

  • $\begingroup$ I was looking for some kind of omnibus test here. Mann-Whitney may be useful as a post-hoc here. $\endgroup$ – Iden Feb 12 '18 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.