# Choice of significant difference test

Im trying to find whether there is a difference between two paired sets of data. The data comes from 20 people and represent test scores before and after taking a course. The sample size is relatively small with only 20 individuals in total. Originally I intended to do a paired T-test, but Im now wondring if Wilcoxon would be better. One last thing Im unsure about is whether to use one or two-tailed test. The values can change in both direction (it can happen - and it did, that for some individuals, the score was actually higher before the course, in most cases it was higher as expected). I used the two-tailed one for this reason.

Here is an example of one of the test results:

Before:

96 77 82 69 94 60 96 97 93 95 65 85 77 97 91 86 96 77 94 84

After:

94 89 86 79 95 62 93 93 95 96 76 82 86 98 94 77 100 82 91 91

The very last thing I would then like to do is to see if there is any difference between the initial and final scores in general between males and females. In this case the test would therefore be unpaired and Im again unsure whether to use one- or two-tailed and T-test or Wilcoxon.

Any help is very much appreciated!

• I really don't see much that would concern me about using either; what's more important is the specific null and alternative hypotheses you're interested in for each case. Commented Sep 14, 2017 at 2:45

Originally I intended to do a paired T-test, but Im now wondring if Wilcoxon would be better.

x1 <- c(96, 77, 82, 69, 94, 60, 96, 97, 93, 95, 65, 85, 77, 97, 91, 86, 96, 77, 94, 84)
x2 <- c(94, 89, 86, 79, 95, 62, 93, 93, 95, 96, 76, 82, 86, 98, 94, 77, 100, 82, 91, 91)

delta <- x2 - x1

hist(delta)


It does not look like the difference between the two are wildly deviating from being normal and I doubt if Wilcoxon Signed Rank will get you any farther. Non-parametric tests have lower power than parametric and if your assumptions are not grossly violated, t-test is fine. If t-test says don't reject the null hypothesis, then Wilcoxon is likely going to say the same. (In this case, the p-value is 0.08875).

One last thing Im unsure about is whether to use one or two-tailed test.

That depends on the nature of the variable and how you see the world operates. One problem is that if you structure it as a one-side test, first, you'll attribute all type I error rate to one side, making you easier to reject the null hypothesis; this is often frowned upon and ii) it'd make you unable to detect if the the intervention is significantly "harmful," which in many situations it may be a valuable piece of information.

The very last thing I would then like to do is to see if there is any difference between the initial and final scores in general between males and females. In this case the test would therefore be unpaired and Im again unsure whether to use one- or two-tailed and T-test or Wilcoxon.

Generally two-tailed because you may want to be able to test if males are significantly higher than females as well as if females are significantly higher than males.

On whether it's one- or two-tailed I'd suggest for new learners stick to two-tailed. Two-tailed is more conservative and if it says reject, one-tailed will also say reject. In most hypothesis tests we often are interested in difference going both directions. As you learn more, you may step into one-tailed tests such as comparing two treatments' non-inferiority or superiority.

Another important point is that you should have decided one- or two-tailed during the planning of the analysis. Do not decide that after you have already analyzed the data. And within you best capability, plan the test ahead and avoid analyzing the same data again and again with different techniques.

Experiment:

$H_0: \mu_1 = \mu_2$

$H_1: \mu_1 \neq \mu_2$

Code:

x1 = c(96, 77, 82, 69, 94, 60, 96, 97, 93, 95, 65, 85, 77, 97, 91, 86, 96, 77, 94, 84)
x2 = c(94, 89, 86, 79, 95, 62, 93, 93, 95, 96, 76, 82, 86, 98, 94, 77, 100, 82, 91, 91)
t.test(x1, x2, paired = T)         # p-value = 0.06807
wilcox.test(x1, x2, paired = T)    # p-value = 0.08875


Result: $H_0$ is not rejected at the 5% significance level.

• Significance does not convey anything about reliability. The Wilcoxon test in your code is not paired and you didn't address one vs two-tailed. If there is reason to assume x1 is less than x2, then both tests would be significant: t.test(x1, x2, paired = T, alternative = 'less'), wilcox.test(x1, x2, paired = T, alternative = 'less') Commented Sep 13, 2017 at 20:06
• I forgot to add the 'paired=T' argument in wilcox.test, I fixed that just now but the result doesn't change. Not sure what you understood by 'reliable conclusion', but for me it's a conclusion that's not bound to change if you were to repeat the same experiment with larger samples. I never said it relates to significance. Commented Sep 13, 2017 at 20:19
• This contradicts the previous statement that the sample size is too low. Commented Sep 13, 2017 at 20:21
• "Your samples are too small to reach any kind of reliable conclusion." I am cautious about this statement as a sample size of 20 may not be something to brag about, whether it's "too small" should be considered together with i) designated type I error and power and more importantly ii) effect size which represents the signal to noise ratio. If the mean difference is dramatic and standard deviation is small, n=20 could have good power and hence be a legitimate sample size. Commented Sep 13, 2017 at 20:23
• FransRodenburg - There is no contradiction, you may have misunderstood something. Anyway, my comment on sample size was a personal judgement call, hence the word opinion right next to it. The reason why one must chose a two-tailed over a one-side test has been explained by @Penguin_Knight in his more thorough answer (which I have upvoted). Commented Sep 14, 2017 at 7:10