How best to compare two processes to know if they perform statistically the same?

Question

I have two processes, A and B. I have tested A and B separately on different tasks, and each performed at a 90% level on their tasks (say, text classification). Now, I pick a random set of 100 tasks to compare A and B, and on the random set, A did better on 60 of the tasks, and B did better on 40 of the tasks. But the difference was only marginal on each. Say, when A "won" it was only better by 1 or 2 "points" and likewise for B when it won.

Is there a way to "score" the results of the processes to know which one is statistically better than the other, or to know they are statistically the same, when they give slightly different results most of the tasks?

Answer 1

This sounds like a situation for paired testing. You have the performance of each technique on the first task, so take the difference in performance. Then do the same for the second task, third task…

This gives you $100$ differences that you can test for significance. If you want to show that the two techniques have equivalent performance, you must first define what you mean by equivalent, but once you do, various forms of equivalence testing could be appropriate, the easiest of which to understand is two one-sided testing (TOST).

In my view, if technique A is better $60$ out of $100$ times and by about the same magnitude as B when B outperforms A, that makes it seem like A is performing better, but you might have a different sense of what constitutes equivalence (which is fine).