Paired categorical data--which test to use

Question

I am trying to determine which statistical test to use for the following:

I used a rubric to determine students' level of achievement on two different tasks (see table below). I want to know if their achievement is statistically different from task 1 to task 2. Do I use a chi square for this? I didn't think I could because the samples are paired. I can't use a paired t-test because the outcome is categorical. Any help?

Answer 1

Let $X$ be the level for Task 1 minus the level for Task 2 for each subject. Then $$X = (-1, -2, -2, -2, -2, -1, 1, 1),$$ ignoring subjects for which no difference was found.

Sign test: A one-sided sign test has P-value about 14%. (For a two-sided test, double this P-value.)

pbinom(2, 8, .5)
[1] 0.1445313

Permutation test: A permutation test can distinguish between differences -1 and -2 and so is more sensitive. This two-sided test has P-value about 12%. [The P-value is simulated, but two subsequent runs with different seeds also gave P-values about 12%.]

set.seed(1117)
x = c(-1, -2, -2, -2, -2, -1, 1, 1)
t.obs = sum(x);  t.obs
[1] -8
t.prm = replicate(10^5, sum(sample(c(-1,1), 8, rep=T)*x))
mean(abs(t.prm) >= abs(t.obs))  # two-sided P-value
[1] 0.11746

The distribution of totals of the sign-permuted differences is shown below. The heights of bars outside the vertical dotted lines show the P-value.

Addendum: For your manuscript, I suggest putting the data into a different format as in the last four rows below, to show how each student performed on each task and with $X_i$ the differences in performances.

stu = 1:14
t1 = c(1,1,1,1,1, 1,1,1,1,1, 2,2,3,3)
t2 = c(1,1,1,1,1, 2,3,3,3,3, 1,3,2,3)
x = t1 - t2
rbind(stu, t1, t2, x)
    [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13] [,14]
stu    1    2    3    4    5    6    7    8    9    10    11    12    13    14
t1     1    1    1    1    1    1    1    1    1     1     2     2     3     3
t2     1    1    1    1    1    2    3    3    3     3     1     3     2     3
x      0    0    0    0    0   -1   -2   -2   -2    -2     1    -1     1     0

If you are using the permutation test, argue that the $0$'s provide no information about different levels of performance on the two tasks. Also mention that this is a paired test and that it is two-sided because you had no reason to believe a particular one of the tasks would result in higher scores. You are testing $H_0: \delta = 0$ and $H_a: \delta \ne 0,$ where $\delta$ is the population difference between the two Tasks.

Perhaps, mention that data are far from normal so a t-test does not seem appropriate and that there are many ties so that the Wilcoxon test may not be appropriate.

The rationale for the permutation test is as follows: Under the null hypothesis we are assuming that T1 and T2 are not different. If that assumption is true it should make no difference whether signs of the $X_i$ are randomly switched. As the data stand, we observe $T = \sum_i X_i = -8.$ Then the question is how $T$ changes under random sign-switching. The answer is that sign-switching can make $T$ either larger or smaller (as shown in the histogram). In fact, $|T|$ exceeds the observed value $|T| = 8$ for about 12% of the permuted values, so that $|T| = 8$ is not a sufficiently remarkable result to be called statistically significant.

Of course, whichever test you use in the manuscript, you need to check everything to make sure that the table in your Question matches the table below. If you are even marginally familiar with R, you can paste the few lines of code into the R 'Console' window and run it to verify the P-value.

If you use an 'exact' version of the Wilcoxon signed-rank test, you need to explain that it is not subject to the famous difficulties with ties that would make the traditional implementation of the test problematic here. Otherwise, some reviewer is going to waste time going back and forth about the validity of Wilcoxon.

Additional R code for the figure, in case you need it:

hist(t.prm, prob=T, br=(-13:12)+.5, col="skyblue2", main="Permutation Dist'n of Totals")
abline(v=c(-7,7), col="red", lwd=2, lty="dotted")

Answer 2

This is tough question. I will try to analyze your data. Your purpose is test if students' performance on task 2 was significantly better than task 1.

Let look at the results of test 1. 10 students were at level 1. These 10 students had no way to get worse at test 2 than at test 1. They would stay at level 1 or get better. So results of test 2 from these 10 students are not random, and they fixed on no change or better.

The same argument is true for 2 students who got level 3 on test 1. They have have no chance to get better results on test 2. Their results on test 2 are getting worse or no change.

So only two students with level 2 at test 1 can provide a information on better or worse in the test 2.

For the students in level 2 at the test 1, who stayed at level 2 at the test 2 would not provide the information on improvement. So we just need to focus on student who raised into level 3 and dropped down to level 1. Under the null hypothesis, the probability of raising or dropping is the same as 0.5. So we just need to use the binomial distribution to test this hull hypothesis. The p value should be $\Pr(X\ge x) = 1 - \Phi(x-1)$, where $\Phi$ is CDF of binomial distribution with $n$ = total # of students with increase or decrease, p =0.5, and x is the number of students with increase. Here the one side test is used because you just consider the improvement.

In you case, one of them got better score and another one get worse score. $p=1- \Phi(0) = 1-0.5*0.5 = 0.75$. So it seems we cannot reject the null hypothesis that performance on test 2 have no improvement over that on test 1.

Answer 3

Your measures are not repeated- they are derived from two distinct assessments. However, there exists some correlation arising from the same persons generating the data. Additionally, the output of the assessments are ordinal ranks and analysis should incorporate this aspect.

I believe that a measure of rank correlation will give you an assessment of whether there are differences between the two assessments. The higher the correlation, the lower the difference between assessments. You have many ties in your data. Try Kendall’s tau-b (using some of BruceET's code):

t1 <- c(1,1,1,1,1,1,1,1,1,1,2,2,3,3)
t2 <- c(1,1,1,1,1,2,3,3,3,3,1,3,2,3)

cor.test(t1, t2, method = 'kendall')

This approach yields a correlation of 0.175 with a p-value of 0.4948. Thus, we cannot conclude that the scores correlate. While this may not be the same as not having differences and in my opinion, one cannot subtract two unrelated measures and remain on solid footing with regard to any experimental design or statistical procedure.

For example, we cannot use any flavor of McNemar's test, as the assessment results are ordered and not simple yes/no trials or nominal categories. Taking the difference between the two different assessments, as suggested for the signed-rank test makes some sense, but ultimately one cannot subtract apples widths from planet widths, for example, and expect to meaningfully compare the differences.

Also, with a little more work, a 95% confidence interval can be obtained via bootstrap for the tau-b statistic:

 boot.tau <- function(data, indices) {
   d <- data[indices,]
   tau.b <- cor(d[,1], d[,2], method= 'kendall')
   return(summary(tau.b)[3])
 }

 df <- data.frame(t1, t2)
 boot.tau <- data= df 
 boot.df <- boot(data=df, statistic = boot.tau , R=1000)
 plot(boot.df)
 boot.ci(boot.df, type = 'basic')

Yielding 95% CI: -0.2671, 0.7050