2
$\begingroup$

So I used a survey, where I had the participants answer 4 Likert-scale questions both before and after a presentation. Ended up with 7 participants total. Since I want to compare their responses pre and post, I had them enter in a 4-digit code before each survey.

I know my data is ordinal, so should I be using a paired Wilcoxon signed rank test?

Would I need to run a separate test for each of the questions, or is there a way to combine them with a test like this?

Improve this question
$\endgroup$
7
  • 3
    $\begingroup$ 7 is a very small sample size. If you run four different tests and correct for multiple testing, you'll have very little power if any at all; in other words, finding significances is unlikely and even if one of the tests is significant, chances are you'll lose this correcting for multiple testing. Are you sure you need significance tests? Can't you just visualise the data and give a qualitative interpretation? $\endgroup$
    – Christian Hennig
    Commented Apr 9, 2022 at 17:01
  • 2
    $\begingroup$ Not knowing what the questions are and the aim of your study, I don't know whether it makes sense to define a single score (i.e. by adding the Likert items up) and test only that. $\endgroup$
    – Christian Hennig
    Commented Apr 9, 2022 at 17:02
  • $\begingroup$ Thank you so much for your reply. I am not sure I need significance tests, in fact, that is probably what is throwing me off so much. The aim of my study is to determine if student veterans' knowledge and awareness of the value of service dogs increased following an educational presentation. $\endgroup$
    – Sarah May
    Commented Apr 9, 2022 at 17:15
  • $\begingroup$ Likert Scale questions include: I am aware of the benefits of service dogs for student veterans; I am knowledgeable regarding the accessibility guidelines (where service dogs are and are not allowed) for student veterans with service dogs on campus; I am knowledgeable regarding the health benefits of service dogs for student veterans; I am knowledgeable regarding the benefits of service dogs for student veterans when participating in everyday activities (school, work, hobbies, etc.). $\endgroup$
    – Sarah May
    Commented Apr 9, 2022 at 17:16
  • 1
    $\begingroup$ There is much discussion of misuses and misinterpretations of significance tests these days, and some suggest to abandon them altogether. Personally I'd probably graph all four questions with answer before, answer after, and lines connecting answers of the same participant, and interpret that qualitatively. Not sure to whom this is presented; some people will expect you to test significance, but running four tests with sample size 7 doesn't look promising (or sensible) to me. $\endgroup$
    – Christian Hennig
    Commented Apr 9, 2022 at 17:37

1 Answer 1

Reset to default
1
$\begingroup$

@ChristianHenning comments "Not knowing what the questions are and the aim of your study, I don't know whether it makes sense to define a single score...." That is the crux of the matter.

Adding or averaging ordinal Likert scores to get an overall survey score is always controversial. However, in a later comment you mention several of the questions on the survey and I think averaging may be reasonable. You should look at several of these scores to see if you feel average scores are meaningful for your purposes.

Perhaps you are averaging Likert-5 scores for the seven subjects, before and after the training. If so, you might have Before b and After a average survey scores as below.

b; a
[1] 0.4 3.1 2.8 3.4 1.8 3.0 1.0
[1] 0.9 3.9 3.1 4.5 3.5 3.9 2.0

In my fictitious data, every one of the seven participants had a somewhat larger average score After than Before. Also, as one would expect, there is a positive correlation in the scores.

cor(a, b)
[1] 0.9332438

On the scatterplot below, all seven points lie above the 45-degree line through the origin.

plot(b,a)
abline(0, 1, col="blue")

enter image description here

A paired Wilcoxon signed rank test in R (essentially a one-sample test on the seven $b - a$ improvements) shows a significant improvement at the 2% level of significance (The P-value $0.01563 < 0.02 = 2\%.$

wilcox.test(b, a, pair=T)

        Wilcoxon signed rank test

data:  b and a
V = 0, p-value = 0.01563
alternative hypothesis: 
 true location shift is not equal to 0

Of course, your real data may give different results than the fictitious data I used for this illustration.

Note: In case you are interested, my fictitious data were sampled in R as shown below:

set.seed(2022)
b = round(4*rbeta(7, 1, 2),1)
i = round(2*rbeta(7, 1, 1),1)
a = b + i
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.