I am trying to determine which statistical test to use for the following:

I used a rubric to determine students' level of achievement on two different tasks (see table below). I want to know if their achievement is statistically different from task 1 to task 2. Do I use a chi square for this? I didn't think I could because the samples are paired. I can't use a paired t-test because the outcome is categorical. Any help?

enter image description here

  • $\begingroup$ Clear definition of "different from task 1 to task 2" is needed. In you table, 6 students have no difference, and 8 student have difference? Then test the null hypothesis that Pr(no difference) =1/3? $\endgroup$
    – user158565
    Nov 17 '18 at 21:29
  • $\begingroup$ @a_statistician You are correct, Pr(no difference)=1/3. What I'd like to know is if students' performance on task 2 was statistically significantly better than task 1 $\endgroup$
    – MathGuy
    Nov 18 '18 at 1:58

Let $X$ be the level for Task 1 minus the level for Task 2 for each subject. Then $$X = (-1, -2, -2, -2, -2, -1, 1, 1),$$ ignoring subjects for which no difference was found.

Sign test: A one-sided sign test has P-value about 14%. (For a two-sided test, double this P-value.)

pbinom(2, 8, .5)
[1] 0.1445313

Permutation test: A permutation test can distinguish between differences -1 and -2 and so is more sensitive. This two-sided test has P-value about 12%. [The P-value is simulated, but two subsequent runs with different seeds also gave P-values about 12%.]

x = c(-1, -2, -2, -2, -2, -1, 1, 1)
t.obs = sum(x);  t.obs
[1] -8
t.prm = replicate(10^5, sum(sample(c(-1,1), 8, rep=T)*x))
mean(abs(t.prm) >= abs(t.obs))  # two-sided P-value
[1] 0.11746

The distribution of totals of the sign-permuted differences is shown below. The heights of bars outside the vertical dotted lines show the P-value.

enter image description here

Addendum: For your manuscript, I suggest putting the data into a different format as in the last four rows below, to show how each student performed on each task and with $X_i$ the differences in performances.

stu = 1:14
t1 = c(1,1,1,1,1, 1,1,1,1,1, 2,2,3,3)
t2 = c(1,1,1,1,1, 2,3,3,3,3, 1,3,2,3)
x = t1 - t2
rbind(stu, t1, t2, x)
    [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13] [,14]
stu    1    2    3    4    5    6    7    8    9    10    11    12    13    14
t1     1    1    1    1    1    1    1    1    1     1     2     2     3     3
t2     1    1    1    1    1    2    3    3    3     3     1     3     2     3
x      0    0    0    0    0   -1   -2   -2   -2    -2     1    -1     1     0

If you are using the permutation test, argue that the $0$'s provide no information about different levels of performance on the two tasks. Also mention that this is a paired test and that it is two-sided because you had no reason to believe a particular one of the tasks would result in higher scores. You are testing $H_0: \delta = 0$ and $H_a: \delta \ne 0,$ where $\delta$ is the population difference between the two Tasks.

Perhaps, mention that data are far from normal so a t-test does not seem appropriate and that there are many ties so that the Wilcoxon test may not be appropriate.

The rationale for the permutation test is as follows: Under the null hypothesis we are assuming that T1 and T2 are not different. If that assumption is true it should make no difference whether signs of the $X_i$ are randomly switched. As the data stand, we observe $T = \sum_i X_i = -8.$ Then the question is how $T$ changes under random sign-switching. The answer is that sign-switching can make $T$ either larger or smaller (as shown in the histogram). In fact, $|T|$ exceeds the observed value $|T| = 8$ for about 12% of the permuted values, so that $|T| = 8$ is not a sufficiently remarkable result to be called statistically significant.

Of course, whichever test you use in the manuscript, you need to check everything to make sure that the table in your Question matches the table below. If you are even marginally familiar with R, you can paste the few lines of code into the R 'Console' window and run it to verify the P-value.

If you use an 'exact' version of the Wilcoxon signed-rank test, you need to explain that it is not subject to the famous difficulties with ties that would make the traditional implementation of the test problematic here. Otherwise, some reviewer is going to waste time going back and forth about the validity of Wilcoxon.

Additional R code for the figure, in case you need it:

hist(t.prm, prob=T, br=(-13:12)+.5, col="skyblue2", main="Permutation Dist'n of Totals")
abline(v=c(-7,7), col="red", lwd=2, lty="dotted")
  • $\begingroup$ Thanks, will a Wilcoxon signed-rank test work for this as well? $\endgroup$
    – MathGuy
    Nov 18 '18 at 1:50
  • $\begingroup$ Corrected version of previous comment: Tried traditional implementation in R. Doesn't seem to be significant at 5%. You might want to try an 'exact' version of the one-sample Wilcoxon test. There are too many ties for the traditional version. (Two-sided, one sample Wilcoxon test in Minitab, tried just now gives P-val 16%.) // Obviously not normal data, but explored t-test also, after putting six 0's back into vector x. Got P-value about 19% for 2-sided, which I don't trust because I think your data pushes the legendary robustness of the t test against non-normality a bit too far. $\endgroup$
    – BruceET
    Nov 18 '18 at 3:13
  • $\begingroup$ I still think the permutation test shown in my Answer is best. Smallest P-value and no problems with assumptions. $\endgroup$
    – BruceET
    Nov 18 '18 at 3:19
  • $\begingroup$ Should the differences resulting zero be deleted? They seem like part of evidence of difference or non-difference. $\endgroup$
    – Todd D
    Nov 19 '18 at 4:25
  • $\begingroup$ The 0's must be deleted for the sign test. And they are irrelevant for the permutation test because they add noting to totals. If you were to stretch robustness against non-normality to do a t-test, then the 0's would have to be included. $\endgroup$
    – BruceET
    Nov 19 '18 at 8:02

This is tough question. I will try to analyze your data. Your purpose is test if students' performance on task 2 was significantly better than task 1.

Let look at the results of test 1. 10 students were at level 1. These 10 students had no way to get worse at test 2 than at test 1. They would stay at level 1 or get better. So results of test 2 from these 10 students are not random, and they fixed on no change or better.

The same argument is true for 2 students who got level 3 on test 1. They have have no chance to get better results on test 2. Their results on test 2 are getting worse or no change.

So only two students with level 2 at test 1 can provide a information on better or worse in the test 2.

For the students in level 2 at the test 1, who stayed at level 2 at the test 2 would not provide the information on improvement. So we just need to focus on student who raised into level 3 and dropped down to level 1. Under the null hypothesis, the probability of raising or dropping is the same as 0.5. So we just need to use the binomial distribution to test this hull hypothesis. The p value should be $\Pr(X\ge x) = 1 - \Phi(x-1)$, where $\Phi$ is CDF of binomial distribution with $n$ = total # of students with increase or decrease, p =0.5, and x is the number of students with increase. Here the one side test is used because you just consider the improvement.

In you case, one of them got better score and another one get worse score. $p=1- \Phi(0) = 1-0.5*0.5 = 0.75$. So it seems we cannot reject the null hypothesis that performance on test 2 have no improvement over that on test 1.

  • $\begingroup$ I agree. It is definitely not significant, but I am trying to figure out how to write it up and which test to use. It is a very small part of a manuscript I am writing. $\endgroup$
    – MathGuy
    Nov 18 '18 at 3:04
  • $\begingroup$ See the test I added into Answer. $\endgroup$
    – user158565
    Nov 18 '18 at 3:22

Your measures are not repeated- they are derived from two distinct assessments. However, there exists some correlation arising from the same persons generating the data. Additionally, the output of the assessments are ordinal ranks and analysis should incorporate this aspect.

I believe that a measure of rank correlation will give you an assessment of whether there are differences between the two assessments. The higher the correlation, the lower the difference between assessments. You have many ties in your data. Try Kendall’s tau-b (using some of BruceET's code):

t1 <- c(1,1,1,1,1,1,1,1,1,1,2,2,3,3)
t2 <- c(1,1,1,1,1,2,3,3,3,3,1,3,2,3)

cor.test(t1, t2, method = 'kendall')

This approach yields a correlation of 0.175 with a p-value of 0.4948. Thus, we cannot conclude that the scores correlate. While this may not be the same as not having differences and in my opinion, one cannot subtract two unrelated measures and remain on solid footing with regard to any experimental design or statistical procedure.

For example, we cannot use any flavor of McNemar's test, as the assessment results are ordered and not simple yes/no trials or nominal categories. Taking the difference between the two different assessments, as suggested for the signed-rank test makes some sense, but ultimately one cannot subtract apples widths from planet widths, for example, and expect to meaningfully compare the differences.

Also, with a little more work, a 95% confidence interval can be obtained via bootstrap for the tau-b statistic:

 boot.tau <- function(data, indices) {
   d <- data[indices,]
   tau.b <- cor(d[,1], d[,2], method= 'kendall')

 df <- data.frame(t1, t2)
 boot.tau <- data= df 
 boot.df <- boot(data=df, statistic = boot.tau , R=1000)
 boot.ci(boot.df, type = 'basic')

Yielding 95% CI: -0.2671, 0.7050

t* and QQ plot for t*


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.