Correct statistical test to compare two results from different algorithms ran on same data set

So I have an algorithm $$A$$ and an algorithm $$B$$ that is run on a set of $$n$$ documents. The documents are financial forms that consist of data that algorithm $$A$$/$$B$$ extracts. For each of the $$n$$ documents, algorithm $$A$$/$$B$$ is run and then these extractions are tested which results in an accuracy measured in %, i.e. $$0 \leq Accuracy(Extraction_i) \leq 100$$ where $$1 \leq i \leq n$$. This results in two arrays/vectors of accuracies, $$Vec_A$$ and $$Vec_B$$.

Let's say that the average accuracy for algorithm $$A$$ is $$x$$% higher than for $$B$$. What statistical test would I use in order to be able to claim that, with 95% confidence, algorithm $$A$$ performs better than $$B$$ for the chosen data set? (The same data set is used for both algorithms).

Thankful for any pointers, been a while now since I took a course in statistics.

1 Answer

It depends on the distribution. If it's close enough to the normal distribution, you can make a paired t-test, otherwise, a Wilcoxon test might be more appropriate. (I'm assuming that both algorithms worked on the same set of documents and that the data is paired.)

Edit:

To assess normality you can use Shapiro-Wilk's test. However, if the sample size $$n$$ is large ($$n > 30$$) you should also look to a Q-Q plot to see if the deviation from normality is small.

• Thanks a bunch. Yes, the data is paired, and both of the algorithms were ran over all the n documents. If I remember correctly, there is some test that you can use for testing whether a given distribution is a normal one or not - could this be used here perhaps? – Nyfiken Gul Apr 25 '19 at 13:43
• I edited my answer to include the normality assessment. – Ertxiem - reinstate Monica Apr 25 '19 at 13:56
• Awesome. Thanks a lot mate! – Nyfiken Gul Apr 25 '19 at 15:20
• You're welcome. – Ertxiem - reinstate Monica Apr 25 '19 at 16:27