1
$\begingroup$

I have a matched paired dataset (student survey before a topic was taught and student survey after the same topic was taught) where I compare the means of their answers (answers are given in Likert scales).

I use the Wilcoxon signed rank test (as opposed to the rank sum test) to compare scores of individual questions to see if they are significant, where my code is

wilcox.test(score ~ time, data = "question 1", paired = TRUE)

I want to test and see if let's say previous experience in the topic was a contributing variable in creating a significant jump in mean from before to after in a group of questions (etc. questions 1-4). But I am stuck on which method of analysis to use in this situation.

I initially used a Kruskal Wallis test, then performed a post hoc Dunn test with Bonferroni adjustment, which gave results such as "test scores of students with no previous experience in a topic were significant from before to after" and "test scores of students with previous experience in a topic were not significant from before to after". But I realized K-W is only appropriate when the variables are independent. So my question is, are my variables dependent because the students are from the same sample and the data are matched this way, or are they independent because my matched data is being compared to previous experience which is an independent variable?

Improve this question
$\endgroup$
1
  • $\begingroup$ Welcome to Cross Validated! Do you have a sense of how you would do it parametrically? $\endgroup$
    – Dave
    Commented Jul 7, 2023 at 2:57

2 Answers 2

Reset to default
1
$\begingroup$

First, a side note: There are no nonparametric tests for comparing means. A mean is an inherently parametric thing. Nonparametric tests primarily test whether responses in one group tend to be larger than responses in another group (stochastic ordering; concordance probability).

A second side note: Regarding the Wilcoxon signed-rank test. This test, unlike the unpaired Wilcoxon-Mann-Whitney two-sample rank test, yields different answers depending on how you transform Y. The rank difference test solves this problem.

But you have come up against a problem. We have flexible ordinal regression models for the unpaired situation, and we have the rank difference and signed-rank tests for the paired situation where there are no covariates to adjust for. But we are lacking in methods for robustly analyzing paired data where other variables need to be considered.

The only approach I can think of, and for which feedback would be much appreciated, is this. Let X be the pre response value and Y be the post response, and Z be another covariate such as previous experience. Fit an ordinal model such as the proportional odds model that generalizes the unpaired Wilcoxon and Kruskal-Wallis methods. Symbolically the model is Y ~ X + Z, which can be extended to Y ~ f(X) + Z + g(X)Z where linearity of X is relaxed by expanding X to multiple columns using spline functions f and g, noting that this would require more data but allows Z to modify how the pre relates to the post. This is an analysis that is conditional on pre. Then from the fitted model we estimate Pr(Y > X | X, Z) for a variety of X and Z values. Thus we are estimating the probability that the post exceeds the pre for all values of pre. Probabilities above 0.5 would indicate increases in Y in going from pre to post.

Another thing to consider: Conditional logistic regression is a generalization of the McNemar test for paired binary data, and may somehow come into play.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I agree with the intuition that means are in some sense "parametric," but this term doesn't feel quite right. There are nonparametric tests for means. For instance, consider the family of all symmetric distributions of finite mean and apply any nonparametric test of medians (which coincide with their means). Although "symmetric" is the kind of strong condition we feel is needed to test means, it's still nonparametric. $\endgroup$
    – whuber ♦
    Commented Jul 27, 2023 at 14:52
  • 1
    $\begingroup$ Yes there's undoubtedly better terminology than nonparametric. I should have left it at "we don't have a nonparametric test of means". A test that assumes symmetry would not quite qualify as nonparametric in this setting. On a slightly related matter empirical likelihood and bootstrap do not work well for means with highly skewed distributions. $\endgroup$
    – Frank Harrell
    Commented Jul 27, 2023 at 15:40
0
$\begingroup$

Conversely to the other persons who already answered you, I am not an stats expert (just an applied researcher). Therefore, my answer will probably be less valuable and more simplistic than those you already have. However, I give it in case it could be useful.

I feel that the analyses you could use depend on how "previous experience" is defined:

-If previous experience is a binary factor (having previous experience vs. no having previous experience) perhaps you could just use a Yuen's test to compare the trimmed means of the experienced/non-experienced groups on a difference score (post-pre survey scores). If you do not like the idea of the difference score, you could also compare the trimmed means of the experienced/non-experienced groups using robust between x within analogs of ANOVA that do not make any distributional assumption. These methods are described in section 8.6 of Wilcox, R.R. (2022) "Introduction to robust estimation and hypothesis testing". 5th edition. Springer.

-If experience is not defined as a binary factor but as a continuous variable, you could probably use robust ANCOVA-like methods that do not make distributional assumptions and can handle curvature. These methods are described in chapter 12 of Wilcox, R.R. (2022) "Introduction to robust estimation and hypothesis testing". 5th edition. Springer.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.