1
$\begingroup$

So I have an algorithm $A$ and an algorithm $B$ that is run on a set of $n$ documents. The documents are financial forms that consist of data that algorithm $A$/$B$ extracts. For each of the $n$ documents, algorithm $A$/$B$ is run and then these extractions are tested which results in an accuracy measured in %, i.e. $0 \leq Accuracy(Extraction_i) \leq 100$ where $1 \leq i \leq n$. This results in two arrays/vectors of accuracies, $Vec_A$ and $Vec_B$.

Let's say that the average accuracy for algorithm $A$ is $x$% higher than for $B$. What statistical test would I use in order to be able to claim that, with 95% confidence, algorithm $A$ performs better than $B$ for the chosen data set? (The same data set is used for both algorithms).

Thankful for any pointers, been a while now since I took a course in statistics.

$\endgroup$

1 Answer 1

3
$\begingroup$

It depends on the distribution. If it's close enough to the normal distribution, you can make a paired t-test, otherwise, a Wilcoxon test might be more appropriate. (I'm assuming that both algorithms worked on the same set of documents and that the data is paired.)


Edit:

To assess normality you can use Shapiro-Wilk's test. However, if the sample size $n$ is large ($n > 30$) you should also look to a Q-Q plot to see if the deviation from normality is small.

$\endgroup$
3
  • $\begingroup$ Thanks a bunch. Yes, the data is paired, and both of the algorithms were ran over all the n documents. If I remember correctly, there is some test that you can use for testing whether a given distribution is a normal one or not - could this be used here perhaps? $\endgroup$ Apr 25, 2019 at 13:43
  • 1
    $\begingroup$ I edited my answer to include the normality assessment. $\endgroup$ Apr 25, 2019 at 13:56
  • 1
    $\begingroup$ You're welcome. $\endgroup$ Apr 25, 2019 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.