1
$\begingroup$

I developed two version of the same algorithm to detect and analyze some medical data through iPhone's camera and I want to compare the result in a mathematical way in order to know which algorith gives better results.

The results of the test can be : SUCCESS, FAIL or INVALID. I tested 56 samples with both the algorithms.

These are the results:

          ALGORITHM 1             ALGORITHM 2
FAIL           12                     14
INVALID        12                      0
SUCCESS        32                     42

Is correct Applying Fisher Test to these data? I had done it with R Studio and I obtained a p-value << 0.05, so I think this show that one of the algorithm Is better than the other, but I'm not sure I can do it. Thanks a lot

Improve this question
$\endgroup$
1
  • $\begingroup$ Are the outcomes ordered? (i.e. is "invalid" less serious - and thereby closer to success - than "fail" is?) ... and if not, what does "better" mean? $\endgroup$
    – Glen_b
    Commented Feb 23, 2018 at 10:59

1 Answer 1

Reset to default
1
$\begingroup$

TLDR: No.

The data is compatible with Fisher's exact test, in your case using the algorithm as the categories and fail/invalid/success as the input. However, what Fisher's Exact test shows is whether the two algorithms give significantly different patterns of categorisation. It is not the appropriate test for your hypothesis of one being 'better'.

see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5426219/ for a useful scientific article explaining Fisher's exact, many more can be found on scientific search engines.

To determine which is 'better' you need to have a ground truth to compare against. I.e. do you know how which samples should be fails, should be invalid and should be success? You need a statistical test based on contingency tables, but which one applies depends on what exactly matters most about the comparison.

Does it matter more it gets fails right? Or maybe successes or even correctly flagging invalid result? Are all of equal importance? I would advise you search around and learn about contingency tables and associated metrics such as accuracy, sensitivity, specificity, positive and negative likelihood ratios, area under the curve etc.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Yes I know in advance the result of the sample. -SUCCESS means: the sample is positive/negative and the algorithm detects positive/negative. -NEGATIVE means: the sample is positive/negative and the algorithm detects negative/positive - INVALID means: the sample is positive/negative and the algorithm can't detect it Which other test do you suggest to me in this case?Thanks $\endgroup$
    – Fab
    Commented Feb 23, 2018 at 11:20
  • $\begingroup$ Hi Fab, In that case looking at the means for success will be the most useful next step. However, the methods in the last paragraph in my answer are probably the more appropriate approach overall. There is not enough info here to provide a useful answer. $\endgroup$
    – ReneBt
    Commented Feb 23, 2018 at 11:28
  • $\begingroup$ Thanks a lot. One last question: Can I say that fisher show me that the two algorithms give significantly different results , so one is probably better than the other BUT to know who is better (because fisher don't give me this information) I will use different methods? For example the means that you suggested before or precision and recall , etc. ? $\endgroup$
    – Fab
    Commented Feb 23, 2018 at 11:57
  • $\begingroup$ That is correct $\endgroup$
    – ReneBt
    Commented Feb 23, 2018 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.