0
$\begingroup$

I did a pre and post-test with a small number of students ($n=12$).These values came after a science intervention that took $4$ weeks. I used a Wilcoxon test for 2 related samples as there was not a normal distribution. And I have gotten some values, but I don’t know how to make sense of the values I got. Simply put, what is the meaning of these values?

The results are shown below:

[Total N    12
Test Statistic  71,500
Standard Error  12,639
Standardized Test Statistic 2,571
Asymptotic Sig.(2-sided test)   ,010]

Pre-Score

Post Score

Q-Q

Q-Q2

$\endgroup$
7
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Nov 14, 2023 at 19:03
  • $\begingroup$ It seems that two groups are very different, which is likely to be what you expect. As you have such a tiny sample, you could show us the data and explain what your variables are, because the interpretation depends on that too. In any case, a plot of data is likely to be more helpful than a test like this and is a vital complement regardless. $\endgroup$
    – Nick Cox
    Nov 15, 2023 at 13:11
  • $\begingroup$ Hello, Nick. I have made some changes that are likely to make things more clear now. Basically, I have one set of student test scores before and after a science intervention. My variable here is their learning gains on a particular science subject. $\endgroup$
    – jordanLeah
    Nov 15, 2023 at 16:58
  • $\begingroup$ Thanks, but what we need are the 12 scores both before and after. If you're comparing those pairs, it is the distribution of the differences that is crucial. Moreover, I see 9 points on each quantile plot not 12. $\endgroup$
    – Nick Cox
    Nov 16, 2023 at 16:31
  • $\begingroup$ Oh, thanks for pointing that out cause I hadn’t noticed. Respectively for 12 participants the changes were from pre to post; 1) 9 to 15, 2) 11 to 13, 3)24 to 26, 4) 13 to 16, 5)12 to 13, 6) 10 to 12, 7) 9 to 10, 8) 9 to 12, 9) 18 to 16, 10)11 to 17, 11) 7 to 8, 12)19 to 20. $\endgroup$
    – jordanLeah
    Nov 16, 2023 at 16:48

1 Answer 1

2
$\begingroup$

I am taking this as an example of the X-Y problem: see here for more. The underlying question is how to summarize and report a comparison for two groups. You have focused on a Wilcoxon test, but my view is that retreating to such a test is too pessimistic: a t test will work fine and plots of the data and a bootstrapped confidence interval are even more helpful.

Here are the data as entered into Stata. People applying other software may find some adaptation of what is below easier for using their favourite than transcribing the OP's comment.

clear
input float(id pre post diff)
 1  9 15  6
 2 11 13  2
 3 24 26  2
 4 13 16  3
 5 12 13  1
 6 10 12  2
 7  9 10  1
 8  9 12  3
 9 18 16 -2
10 11 17  6
11  7  8  1
12 19 20  1
end

The data are integer scores. For a paired t-test, it's the distribution of the differences that is crucial, not the distribution of the scores in each group. For such data, normality is an ideal condition (often, but misleadingly, described as an assumption). There is a judgment call on whether this is satisfied: my call, like anybody else's, is based partly on experience. It seems an adequate approximation for such data. A check is that the equivalent Wilcoxon test yields a similar if less transparent result. I don't feel compelled to be slavish to any formal test of normality or to say skewness and kurtosis summaries, although the latter are supportive.

FWIW, here is a normal quantile plot of the differences. (I still don't understand why there are only 9 points on the OP's quantile plots.)

enter image description here

A t-test yields a P-value of 0.0059 for a two-sided test. A one-tailed test could be done too. What seems more interesting, more useful, and more robust is a bootstrap confidence interval for the difference of means:

. estat bootstrap, all

Bootstrap results                               Number of obs     =         12
                                                Replications      =       1000

      Command: summarize diff, meanonly
        _bs_1: r(mean)

------------------------------------------------------------------------------
             |    Observed               Bootstrap
             | coefficient       Bias    std. err.  [95% conf. interval]
-------------+----------------------------------------------------------------
       _bs_1 |   2.1666667   .0081667   .59729582    .9959884   3.337345   (N)
             |                                       1.083333   3.416667   (P)
             |                                              1   3.333333  (BC)
------------------------------------------------------------------------------
Key:  N: Normal
      P: Percentile
     BC: Bias-corrected

A simple take-away is that most people improved, consistently with those results with a mean improvement of about 2.2 and 95% confidence interval from 1.0 to 3.3 or from 1.1 to 3.4, depending on what you choose.

Here is a plot of (post MINUS pre) versus pre. Some people might prefer a graph versus (pre PLUS post) / 2. There is a hint of a regression effect with bigger improvements from worse scores first time around.

enter image description here

All that said, this is a tiny sample and should not be over-interpreted.

$\endgroup$
7
  • $\begingroup$ You need to defend your assignment, not me. The most powerful argument to me is that t-test, confidence interval, descriptive statistics and graph indications are all consistent. I suspect the Wilcoxon test is consistent, but it's really about quite a different question and not about comparing means at all. $\endgroup$
    – Nick Cox
    Dec 5, 2023 at 21:32
  • $\begingroup$ Of course. I appreciate your guidance, and will go through/re-assess the same steps to gain a clearer perspective. As you mentioned earlier, I might have been overly fixated on what was taught as the 'normality assumption.' That’s because, I was initially advised to perform a Wilcoxon test due to concerns about the data not having a normal distribution. $\endgroup$
    – jordanLeah
    Dec 6, 2023 at 9:04
  • $\begingroup$ One new question; If based on different perspectives performing a t-test is not accepted due to the “normality assumption”, what more I can do with Wilcoxon values? Could I still perform CI and regression to get into detail about student improvement? How do I do that? $\endgroup$
    – jordanLeah
    Dec 25, 2023 at 13:42
  • $\begingroup$ Looks like the same question to me. I don’t have a different answer. $\endgroup$
    – Nick Cox
    Dec 25, 2023 at 17:08
  • $\begingroup$ I meant to say I can’t perform a t-test because it’s not accepted, could I still report CI and regression with Wilcoxon? And, I perform them on both scores right? – $\endgroup$
    – jordanLeah
    Dec 27, 2023 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.