Can I use Kruskal Wallis test to analyze ranked data? If not, what analysis method should I use? I am currently conducting a survey study for a project. Unfortunately, I would not at all consider myself proficient in statistics, and am struggling with choosing the appropriate analysis method for my study.
I wish to analyze these variables:
(1) IV: A categorical (nominal) demographic variable with 3+ groups (e.g. year level in school)
(2) DV: A variable part of a ranking question, such as a question that asks respondents to rank 5 criteria (A, B, C, and D) for choosing a tutor, from most preferred (1) to least preferred (5).
I'd like to know if there are differences between the year levels when it comes to their preference for a particular ranked item... for example, if first year students differ in how they rate "Criteria A" compared to second year students.
Would it be possible to use the Kruskal Wallis test? I'm hesitant to push through with it, as I could not find any examples using a ranking question as a dependent variable.
Thank you in advance!
 A: You ask specifically about using the Kruskal-Wallis test. It may be useful for your data, as for my simulated data below. Also, it may be useful to look at some of the 'Related' pages on this site, listed in a margin. [However, see Note 2 at the end, for a brief demonstration
of a possibly appropriate chi-squared test of score counts. Without seeing your data, I cannot say which test I would use for them.]
Roughly speaking, the Kruskal-Wallis test is an extension of the two-sample Wilcoxon (rank sum) test,
to accommodate $g > 2$ groups with non-normal
continuous values from populations of roughly the same
shape.

*

*The 'continuous' assumption anticipates no tied observations between or within the group observations.


*Also, for the kind of data you envision, the shapes
of the distributions of the ordinal observations
may be different. This makes it difficult to treat
tests results just as looking at differences in group medians.
Nevertheless, when the K-W test gives significant P-values, it is often possible to interpret the result
as showing that groups are different.
Here is an example with $g = 4$ groups of artificial
data simulated in R.
set.seed(202)
x1 = sample(1:5, 20, rep=T, p=c(1,1,2,2,3))
x2 = sample(1:5, 20, rep=T, p=c(1,2,2,3,3))
x3 = sample(1:5, 20, rep=T, p=c(2,2,3,3,1))
x4 = sample(1:5, 20, rep=T, p=c(3,3,2,2,1))

x = c(x1,x2,x3,x4)
gp = rep(1:4, each=20)
boxplot(x ~ gp, col="skyblue2", pch=19)


kruskal.test(x ~ gp)

        Kruskal-Wallis rank sum test

data:  x by gp
Kruskal-Wallis chi-squared = 26.816, df = 3, 
 p-value = 6.433e-06

With appropriate protections (such as Bonferroni adjusted significance levels) against 'false discovery'
it may be appropriate to look ad hoc at some pairs
of differences.  For example, even though the shapes of the distributions for Groups 1 and 4 are differ, it is clear that the scores in Group 1 tend to be larger than those in Group 4.
wilcox.test(x1, x4)

        Wilcoxon rank sum test 
        with continuity correction

data:  x1 and x4
W = 354, p-value = 1.697e-05
alternative hypothesis: 
 true location shift is not equal to 0

Warning message:
In wilcox.test.default(x1, x4) : 
 cannot compute exact p-value with ties

The P-value, even though not exact, is far below the 1%
level. Furthermore, the boxplots above and the empirical CDF (ECDF) plots below, show how the the two groups differ. Scores in Group 1 stochastically dominate (tend to be larger than) those in Group 4.
Specifically, in the plot below, the ECDF for Group 1 (blue) lies to the right of (hence also below) the ECDF for Group 4 (brown).
hdr="ECDF of Gp 1 (blue) Dominates ECDF of Gp 4"
plot(ecdf(x1), col="blue", main = hdr)
 lines(ecdf(x4), col="brown", pch = "o")


Notes: (1) Similarly, after the K-W test, one can show a significant difference between Groups 1 and 3 (with a warning message about ties), but not between Groups 2 and 4.
wilcox.test(x1, x3)$p.val
[1] 1.8489e-05
...
wilcox.test(x2, x4)$p.val
[1] 0.1188832
...

(2) Another possible test is a chi-squared test of the null hypothesis (strongly rejected here) that Groups have similarly distributed scores. Because counts are small, it is necessary to simulate the P-value of the chi-squared test (argument sim=T in R).
TBL = rbind(tabulate(x1), tabulate(x2),  
            tabulate(x3), tabulate(x4))
TBL
     [,1] [,2] [,3] [,4] [,5]
[1,]    0    1    2    6   11
[2,]    3    4    1   10    2
[3,]    7    6    2    4    1
[4,]    1   11    4    3    1

chisq.test(TBL, sim=T)

    Pearson's Chi-squared test 
    with simulated p-value 
    (based on 2000 replicates)

data:  TBL
X-squared = 46.069, df = NA, 
  p-value = 0.0004998

