So I have an algorithm $A$ and an algorithm $B$ that is run on a set of $n$ documents. The documents are financial forms that consist of data that algorithm $A$/$B$ extracts. For each of the $n$ documents, algorithm $A$/$B$ is run and then these extractions are tested which results in an accuracy measured in %, i.e. $0 \leq Accuracy(Extraction_i) \leq 100$ where $1 \leq i \leq n$. This results in two arrays/vectors of accuracies, $Vec_A$ and $Vec_B$.

Let's say that the average accuracy for algorithm $A$ is $x$% higher than for $B$. What statistical test would I use in order to be able to claim that, with 95% confidence, algorithm $A$ performs better than $B$ for the chosen data set? (The same data set is used for both algorithms).

Thankful for any pointers, been a while now since I took a course in statistics.


It depends on the distribution. If it's close enough to the normal distribution, you can make a paired t-test, otherwise, a Wilcoxon test might be more appropriate. (I'm assuming that both algorithms worked on the same set of documents and that the data is paired.)


To assess normality you can use Shapiro-Wilk's test. However, if the sample size $n$ is large ($n > 30$) you should also look to a Q-Q plot to see if the deviation from normality is small.

  • $\begingroup$ Thanks a bunch. Yes, the data is paired, and both of the algorithms were ran over all the n documents. If I remember correctly, there is some test that you can use for testing whether a given distribution is a normal one or not - could this be used here perhaps? $\endgroup$ – Nyfiken Gul Apr 25 '19 at 13:43
  • 1
    $\begingroup$ I edited my answer to include the normality assessment. $\endgroup$ – Ertxiem - reinstate Monica Apr 25 '19 at 13:56
  • $\begingroup$ Awesome. Thanks a lot mate! $\endgroup$ – Nyfiken Gul Apr 25 '19 at 15:20
  • 1
    $\begingroup$ You're welcome. $\endgroup$ – Ertxiem - reinstate Monica Apr 25 '19 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.