0
$\begingroup$

I'm implementing a search engine in two different ways A and B, and I am comparing their accuracies.

Accuracy of system A is measured as follows: Given a query, I check if system A gives me the result that I want. Accordingly, I give a binary score(0 for No, 1 for Yes) I make 100 such queries. The accuracy is just the number of queries in which I was shown the result I wanted.

Similarly, I measure the accuracy for system B. Note that I make the same 100 queries for A and B.

The accuracies of A and B were 25% and 65% respectively. How do I decide if this improvement is not due to random chance, and is because B is better?

$\endgroup$
2
  • $\begingroup$ Hello, welcome to Stats.SE. How much do you know about "paired sample tests", and do you care about the mean accuracy number itself (i.e. does your communication involve "B is x% better than A in terms of accuracy?") $\endgroup$
    – B.Liu
    Jul 3 at 8:46
  • $\begingroup$ I do not know much about them, except that they involve differences between sets of observations. In this case, the possible differences would just be {-1,0,1}. Would that cause issues? And no, my communication does involve the mean accuracy. But I'm curious, how would the answer change if it did not involve the mean accuracy? $\endgroup$
    – jj050102
    Jul 3 at 9:25

1 Answer 1

0
$\begingroup$

Given your samples (queries) are dependent between systems A and B (more precisely, they are paired between the two systems), a test for paired data would be appropriate.

Examples of test that uses paired data include the paired t-test and the sign test. For 0-1 binary responses, the sign test is known as the McNemar test.

The paired t-test is parametric, meaning that you can draw a conclusion on the difference in the accuracy (which is the mean of the 0-1 binary scores) achieved by the two systems. Technically, it assumes you have i.i.d. normal data, which your data certainly is not. Having that said, given your sample size (100), mean outcome in both groups (25%/65%), and the difference between the mean outcomes, the bias incurred from deviating from the normal assumptions shouldn't affect the test outcome.

The sign test is non-parametric, which means you are not hypothesising anything about the accuracy of systems A and B, but whether a system gives you larger (or smaller, or just different) values than the other system in general. Strictly speaking, this mean you should not draw any conclusion on the difference in the accuracy achieved by the two systems from the sign test. Nonetheless, a lot of people do that in practice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.