2
$\begingroup$

I am rather stressed out about my methodology. For my PhD I ran a year long literacy intervention (small scale longitudinal study) that was supposed to test the rate of change in individual students' literacy skills over the year. I chose an individual rate of change per student because I cannot compare students with different socio-economic backgrounds and from different schools etc.

My sample was roughly 90 (3 classess = small-scale). The students had to write multiple writing drafts throughout the year-long intervention so that I could track their changing literacy scores. I numbered the literacy assignments from 1 to 10 and then at the end when I did the analysis I looked at the literacy scores pre-intervention and post-intervention (assignments 1 and 10) for an overview of whether students developed better literacy skills. But the strength of my study (or so I thought, and my supervisors) was in the process/based analysis. I compared the literacy scores for assignment 10 & 9; 9 & 8; 8 & 7 etc to get a detailed analysis of what was happening during the intervention. I wanted to know if there was a better literacy score between assignment 10 & 9; a better score between 9 & 8 etc to see if progress was being made, each one building on the one before. I wanted more than just a pre and post score because often things happen during the year that impact on teaching and learning and we wanted to see if they impacted on development, where in the year to get an idea of what aspect of the syllabus was not working. So I used a Wilcoxon signed rank test and now that I am trying to publish I am being told it was the wrong test to use. Out of 4 reviewers, 3 are happy with the Wilcoxon but one said it should have been Kruskal-Wallis. I am being told that because I looked at pairwise comparisons over a year, and not just a once off pre and post, I have inflated the error? I am confused and would love some help if possible.

$\endgroup$
  • $\begingroup$ I'm not quite sure what you did. Did you do one Wilcoxon test for assignments 10 vs 9, then another for 9 vs 8 etc, or did you somehow combine all the comparisons into one test? Also, what do you mean by "3 classes"? How do the classes enter the analysis? $\endgroup$ – Gordon Smyth Sep 11 '17 at 0:01
  • $\begingroup$ I just gave context about the sample with the 3 classes so the 90 students came from 3 separate classes. I taught a section, students responded with a writing task, I taught some more, they responded again and this process carried on for the year. Each writing response was given a task number hence 1 - 10. All I wanted to check was whether the students showed an increase in their literacy scores as we progressed so I was told to use the wilcoxon to test matched pairs - was task 10's score greater than task 9. Was task 9's score greater than task 8 etc all the way down to task 1. $\endgroup$ – Tracey Bunn Sep 11 '17 at 0:20
  • $\begingroup$ Sorry, forgot to add. I think we ran a separate wilcoxon signed rank test for each pairwise comparison for all 90 odd students. $\endgroup$ – Tracey Bunn Sep 11 '17 at 0:22
  • $\begingroup$ You need to be sure! Did you run the analysis yourself? $\endgroup$ – Gordon Smyth Sep 11 '17 at 0:35
  • 1
    $\begingroup$ Kruskal Wallis doesn't make sense because it ignores the student effect. $\endgroup$ – Glen_b Sep 11 '17 at 2:35
1
$\begingroup$

In my opinion, the Wilcoxon signed rank tests are fine. It's not the only way you could analyse this data, but it's ok. You need to present nine test p-values in your manuscript, one for each increment (2 vs 1, 3 vs 2, ..., 10 vs 9.).

Your data is not in any sense a one-way anova so the Kruskal-Wallis test is inappropriate. A Kruska-Wallis test would assume that all observations are independent, whereas repeat observations on the same student are related.

The Wilcoxon signed rank test correctly accounts for the fact that observations are paired by student by making a pairwise comparisons.

$\endgroup$
  • $\begingroup$ Thank you Gordon. I did have separate p values for all pairwise samples which enabled me to check if each of the paired scores (the 'changes') were statistically significant. I was very concerned when a journal reviewer came back and said I should not have used a wilcoxon but rather the Kruskal-Wallis (despite using the same student's pairwise comparisons). $\endgroup$ – Tracey Bunn Sep 11 '17 at 0:57
  • $\begingroup$ As you publish more papers, you'll find that reviewers do make mistakes. BTW, if you think I've covered everything that was concerning you, you could mark my answer as "accepted". Or just wait for other answers if you want more opinions. $\endgroup$ – Gordon Smyth Sep 11 '17 at 6:19
1
$\begingroup$

Wilcoxon signed rank tests seem appropriate since they reflects the fact that measures are taken repeatedly on the same subjects (which increases the tests power to detect real effects) but are modest enough to recognize that grades on writing assignments are only ordinal, but not interval variables (i.e. the difference between a B+ and an A could be much smaller than the difference between an A and an A+ etc.).

Doings the Wilcoxon signed rank tests assumes, that even if the steps between grades are not uniformly high, you as a grader would know for any two evolutions you compare among the students which step is bigger of the two. If you cannot do that, you cannot rank the changes and without a ranking there will not be a rank test. You would be limited to a sign test: basically just counting how many students improved, how many stayed put and how many deteriorated. If there are many more improvements than deteriorations, your test will be significant. Such a test is obviously less powerful since it has no notion of how large any of the improvements were. I do not think you need to use this one. If only you can establish a ranking of improvements, you don't.

If on the other hand your literacy score is much more objectively countable like for example counting the number of mistakes per 100 words (I'm no expert in the field, but you see what I mean with objective I believe), then you can even use paired t-tests. They will have higher statistical power to detect real effects.

When used in the right conditions as described above, the power of the tests to detect existing changes compares as follows:

$$\text{t-test} \geq \text{wilcoxon signed rank test} \geq \text{sign test}$$

In any of the three options, use two sided tests since the possibility that a training session might have deteriorated performance is real. That is what everybody does. Doctors also hope their medication works better than a placebo, but they use two sided tests because it might be even worse than doing nothing. (One sided tests would just make your $\alpha$ level less stringent and are frowned upon.)

Just to be sure, these 3 school classes only exist so that you can get a big enough sample size? You have not chosen the three classes to purposefully represent for example one posh private school, one average school and one underprivileged school? If yes, you will need a more complex statistical methodology to include that information in your analysis as well.

Now to the most important part: The preceding caveats and options notwithstanding, you still need to control your p-value cutoffs for multiple testing. It is very important not to confuse two concepts here:

  • your tests are paired on the student level since you observe the same student multiple times (as opposed to observing a different class of 90 students each time after one of your training intervals)
  • your tests are pairwise since you compare multiple intermediary situations (as opposed to only comparing the before-after states)

Being paired is taken care of by signed rank tests (or paired t-tests or sign tests), being pairwise requires the following additional precautions:

When the null hypothesis is true and there is no real effect, you still have the possibility of a false discovery proportional to the $\alpha$ cutoff that you compare your p-value to. That is true for every test, so if you do enough tests, you are bound to find some significant results that are false. You need to correct for this inflated chance of false discovery. The easiest way is the Bonferroni correction, just divide your $\alpha$ level by the number of tests. For example, you would be comparing all your p-values against a cutoff of $\alpha/9=0.55\%$ because you perform 9 tests instead of the usual $5\%$ cutoff for a single test. You can see that this is quite a stringent restriction. Holm's method will be a little bit less stringent while still not inflating the chance of false discoveries, it is preferable for that reason.

Practically speaking, are you sure your results are actionable? If you find out for example that the first two training intervals didn't help, but the third and fourth did help, then the fifth and sixth did active harm, the seventh was neutral again and the last two helped, can you translate such mixed results into actionable recommendations? Recommending to skip the intervals 5 and 6 and to put more emphasis on intervals 3 and 4 is only actionable if each interval did something different with the students. If you can explain based on some theory (not only the statistical data) why it could be that some training sessions helped but others didn't, that's insightful. If all the intervals were supposed to do more of the same but didn't, this will be hard to make sense of.

Also, even if you do the 9 intermediary comparisons, you can still also do a before-after comparison. You just need to adjust your cutoffs for one more test (10 instead of 9)

$\endgroup$
  • $\begingroup$ (+1) But you'd perform the Wilcoxon signed-rank text by calculating the change in each student's grade from one assignment to the next before ranking the magnitude of those changes. So is going from a B+ to an A a bigger or smaller improvement than going from an A to an A+, or is it the same? For that matter is it the same magnitude of change as going from an A to a B+? If you really wanted to be completely non-committal about all that you'd use the sign test. $\endgroup$ – Scortchi Sep 11 '17 at 9:41
  • $\begingroup$ I assumed that the grader would know which grade improvement has which value even if the steps are not uniformly high. I will clarify. $\endgroup$ – David Ernst Sep 11 '17 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.