For a classification task, I have developed two methods and FScore(harmonic mean of precision and recall) of both the classes serves as the performance criteria. How can I check whether the difference between the performance of the two systems is statistically significant or not.

For eg. I was reading a paper by Navigli and in the evaluation he cites that "all the differences between 2 systems are statistically significant (p < 0.01)"

I would like to know how to perform this test of statistical significance.

EDIT : No clue is there regarding which test is been talked about. Is there some standard test used in NLP community? (the work is related to Natural Language Processing)

  • 1
    $\begingroup$ What makes you think "the test he used is called p-test" - there's no indication in the text near the phrase "all the differences between" to suggest what test was used; only a p-value is quoted. When you say 'fscore' what are you referring to? $\endgroup$
    – Glen_b
    May 18 '13 at 7:21
  • $\begingroup$ Best to email the author directly. $\endgroup$
    – Nick Cox
    May 18 '13 at 16:00
  • $\begingroup$ Have you received any answer? I'm actually having the same problem here... $\endgroup$ May 31 '13 at 11:28
  • $\begingroup$ @JoachimBingel : No. I haven't got any definite answer yet. $\endgroup$
    – damned
    May 31 '13 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.