I need to do a statistical test for one of my study but I’m not sure which test is appropriate.

This is my experiment:

I measured performance of 30 tools (F1 scores) on classifying a pair of documents as similar or not. Then, I applied an enhancement process to every tool and measured the tools performance again (they are the same set of tools before and after the process). I found that the enhancement can increase the F1 scores of every tool. However, I would like to test if the performance "improvement" has statistical significance. The 30 tools work deterministically.

I read from a few places that it would be a either a parametric paired t-test or a non-parametric Wilcoxon test depending on the normality of my data. So I tried using Shapiro–Wilk test on both the before and after data, they're not normally distributed (p-value = 5.177e-05 and 0.008338).

So should I go for Wilcoxon test? If not, could you please recommend a test that I should use?


Paired :-)test because you're measuring the same group of documents. Is this numerical or categorical data?

I'm sorry, I believe not paired because the two types of analysis you're comparing are not related. A simple t test with pooled variance should work, unless you believe the two sets of data have a different variance.

  • $\begingroup$ Your two different answers have been edited together. Please edit your answer to choose a single position, but offer better support for whichever it is you now think is correct. $\endgroup$ – Glen_b -Reinstate Monica May 30 '17 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.