2
$\begingroup$

I carried out a test on a group of 100 people. The test contained 10 questions with 5-point Likert scale answers (1 = Strongly Agree, 5 = Strongly Disagree). After the test was conducted, the group was educated over a period of time on how to answer the questions. After the education completed, the same test was carried out again on the same group, but with 10 absent. Knowing that the pre and post tests were anonymously conducted, I would like to test whether there is a significant difference between the two test results. If I use the Wilcoxon Signed Rank Test for a paired sample to compare results for individual question of each test, the results will be different depending of the entry of the test results. In other words, I can't tell if one student improved because I can't link his first test to his second test.

I don't know if using the Wilcoxon Signed Rank Test for a paired sample would be possible given the tests were done anonymously.

Any help is appreciated. Thanks

$\endgroup$
  • $\begingroup$ Values from each question were counted together? For example: If all questions (n = 10) were answered by 5 (strongly disagree) it gives 50 points for particular test? $\endgroup$ – Ladislav Naďo Nov 2 '13 at 20:13
  • 1
    $\begingroup$ If you've lost the pairing you can't pair them! Usually in the situation of having anonymous tests there are some temporary IDs assigned purely for the purpose of matching the pairs of before and after answers (they're not associated with individual information about the person except so far as to enable that matching). If something obvious like that wasn't done, you simply don't have the pairing, and have no way to do it post-hoc (and in the process, whoever designed the experiment has thrown away what might be a lot of variance reduction in the process). $\endgroup$ – Glen_b Nov 2 '13 at 21:00
  • 2
    $\begingroup$ It's perfectly simple to maintain anonymity while matching responses. Just as one example off the top of my head (there will be better procedures than this; it's a common issue solved many different ways), during the first test, you have pairs of cards which you give to each participant (and whose purpose you explain! They can even be drawn randomly from a bag by the participants). They attach one card to their first test and retain the matching second card to attach to the second test. You never find out who is who. If some cards are lost or forgotten, there's overlapping samples tests. $\endgroup$ – Glen_b Nov 2 '13 at 21:11
  • 1
    $\begingroup$ (I missed the online bit ... but that actually isn't really a barrier to assigning IDs either.) Why would you use the signed rank test if you could pair them and the t-test if you couldn't? If you're prepared to make the assumptions of a t-test, why wouldn't you do the same on the paired data? If you were hot prepared to make the required assumptions for a paired t-test before, why would you make those assumptions now that you can't pair? I can't fathom the reasoning here. $\endgroup$ – Glen_b Nov 2 '13 at 21:14
  • 1
    $\begingroup$ @jbowman: What if the intrasubject correlations are negative? Wouldn't the unpaired choice then be slightly anticonservative? I know it's pathetic. $\endgroup$ – Michael M Nov 2 '13 at 23:07
3
$\begingroup$

The short answer, in three parts, is a) no, you can't do a paired test, as has been pointed out in comments, b) yes, you can do an unpaired test, and c) that 10% of non-respondents to the second test may be important.

Let us consider a simplistic hierarchical model of response, where there is an individual-level characteristic $\theta_i$ which has some distribution $f(\theta)$ and a test-specific response $y_{ij}$ for tests $j \in {1,2}$ that depends upon the individual-level characteristic through its distribution $p_j(y_{ij} | \theta_i)$. If we know $i$ for each $y_{ij}$ we can obviously do the paired test, and the difference between $y_{i1}$ and $y_{i2}$ obviously are not influenced by the differences between the $\theta_i$, since it is for the same $i$.

If, on the other hand, we don't know the individual $i$, we are faced with draws from two distributions $p^*_j(y_{ij}) = \int_\Theta p_j(y_{ij} | \theta_i) f(\theta_i)\text{d}\theta_i$. The scores $y_{ij}$ are still independent across $i$ and, under the null, independent across $j$ as well. The distribution itself no longer varies across $i$. Under alternative hypotheses, the distributions $p_j$ will still differ across $j$, it's just that they are population-level distributions rather than individual-level distributions.

Consequently, we can still perform an (unpaired) test for differences between $j$, but it's going to be less powerful than if you could get rid of the extra variability introduced by not knowing the individuals. It's just a matter of what you can condition on; more conditioning reduces variability and thereby increases power.

Personally, I'd use the unpaired version of the Wilcoxon, as you can't lose much relative to the unpaired version of the $t$-test and you might gain a lot. See this question for a little more information.

Of greater concern is that missing 10% of the original sample. You'd really like to understand the missing data mechanism, if any. Consider the possibility that the 10 who dropped out were among the poorest performers on the original test, and that the amount of improvement was strongly negatively related to how well an individual performed on the first test (i.e., poor performers improved a lot more on average than good performers). That, combined with regression-to-the-mean effects, means you'd likely be missing data on some of your largest gains, thus a) weakening your ability to detect a significant difference, and b) biasing your estimate(s) of how much improvement there was downwards. OTOH, under the null hypotheses, we expect to see a gain, because we are including the low scorers in sample 1 but removing 10 likely low scorers from sample 2. So there's an upwards bias there too. Which effect dominates isn't clear, but what is clear is that your test and associated estimates would almost certainly be biased.

For example, if I simulate from the simple model above assuming $\theta_i \sim \text{N}(0,1)$ and $y_{ij} \sim \text{N}(\theta_i,1)$, and drop the $y_{i2}$ for which $y_{i1}$ was in the 10 lowest values, the expected value of $y_{i2} \approx 0.136$ while that of $y_{i1} = 0$. 0.136 is about 0.96 standard deviations above 0, relative to the std. dev. of the difference between the means of $y_{i1}$ and $y_{i2}$, which would obviously have a huge impact on your type I and type II error probabilities.

e2 <- rep(0,100000)
for (i in 1:100000) {
   theta <- rnorm(100)
   y1 <- rnorm(100, theta)
   y2 <- rnorm(100, theta)
   y2[order(y1)[1:10]] <- NA
   e2[i] <- mean(y2, na.rm=TRUE)
}
mean(e2)
[1] 0.1360364
$\endgroup$
  • $\begingroup$ Thanks @jbowman for the detailed answer. Do you refer to Mann-Whitney by the unpaired version of Wilcoxon? It is still unclear to me how I would "validate" the significance, if any. $\endgroup$ – Bran Nov 3 '13 at 0:56
  • $\begingroup$ 1) Yes, they are just different versions of the same test. 2) I think, insofar as validating the significance of the test is concerned, the only issue is the missing data. If you look at the scores on test 1 of the nonrespondents on test 2, that may help you decide whether to consider the missingness to be "at random" with respect to the responses. (Maybe it won't, also.) Otherwise, you can do the analysis, but put a caveat in about the missing data - "this analysis assumes the nonrespondents can be treated as missing at random" or some such. $\endgroup$ – jbowman Nov 3 '13 at 0:59
  • $\begingroup$ Let's assume that I was able to reduce the missing records to a certain ratio (say 1%), what implication would that have on the results? Could it then be ignored? $\endgroup$ – Bran Nov 3 '13 at 1:06
  • $\begingroup$ Well, you can't, speaking mathematically, tell from the ratio alone, as we can always construct corner cases where all of the benefit would have been seen by just the 1% who dropped out. Practically speaking, though, if you got it down to 1-2%, I doubt it would mess up your results enough to matter... maybe if your test was at the 95% level of confidence and it wound up with a p-value of 0.048, you'd have some doubts, but it's also not like the difference between p-values of 0.048 and 0.062 is statistically significant itself. I'd write in a minor caveat and just do the test, myself. $\endgroup$ – jbowman Nov 3 '13 at 1:31
  • $\begingroup$ @jbowman, may I ask your opinion (because I'm not sure myself)? Is it right to say that an unpaired test is asymptotically equivalent to averaging results of the great many paired tests (of the corresponding type) with random pairing? Or maybe of all possible pairings gone over? $\endgroup$ – ttnphns Nov 3 '13 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.