2
$\begingroup$

According to Manning et al. (p. 155) accuracy is the sum of the diagonal in the confusion matrix divided by the sum of all items. On the other hand, following Artstein and Poesio (p . 558) precisely this seems to be also called agreement (without chance correction).

Is there any conceptual difference?

Improve this question
$\endgroup$
1
$\begingroup$

EDITinfo: "accuracy" for multiclass was changed according to Moens (2006, p. 183)

EDIT: Sometimes (seemingly often) agreement and accuaracy are computed similarly. However, they seem to be used for different fields. Agreement is typically for inter-annotator tasks, while accuracy is the basic standard for classification (and other statistical tasks) performance assessment.

Agreement between coders in order to achieve reliability according to Artstein & Poesio (2008, p. 558) "the percentage of judgments on which the two analysts agree when coding the same data independently"

                       Coder 1
                 +-------------+
                  |   P  |  N   |                       
              +---+------+------+
              | P |  tp  |  fp  |                           
    Coder 2   +---+------+------+
              | N |  fn  |  tn  |                       
              +---+------+------+

SUM_all      = sum of all elements (sometimes called items)
NO_classes   = cardinality classes
SUM_diagonal = the sum of all elements on the main diagonal

TWO classes    Agreement =    (tp + tn) / SUM_all    (SIMILAR to accuracy)
i>2 classes    Agreement = SUM_diagonal / SUM_all    (NOT similar to accuracy)

EDIT: Accuracy of classification in order to measure the classification effectiveness (in context of classification denoted as performance -- qualitative) is computed at least two variants (1) Moens (2006, p. 183) accuracy AND (2) Sokolova & Lapalme (2008, p. 430) average accuracy) 

                     Real Class
                  +-------------+
                  |   P  |  N   |                       
              +---+------+------+
 Hypothesized | P |  tp  |  fp  |                           
    Class     +---+------+------+
              | N |  fn  |  tn  |                       
              +---+------+------+


TWO classes             Accuracy =                      (tp + tn) / SUM_all     (SIMILAR to agreement)
i>2 classes (1)         Accuracy =                   SUM_diagonal / SUM_all     (according to Moens (2006, p. 183))
            (2)  averageAccuracy = SUM_i[(tp_i + tn_i) / SUM_all] / NO_classes  (not similar to Agreement or Accuracy (acc. to Sokolova, 2008, p. 430))
                                          ^^^^^^^^^^^^
                                          Note this are calculated relatively for each i
                                          => if the matrix is larger then 2x2 the tn are
                                             ALL other elements in the matrix
                                             which are  NOT associated with the i-class
                                             (i.e. from all other classes -- not only on the diagonal)!

Artstein & Poesio (2008) Inter-Coder Agreement for Computational Linguistics http://www.newdesign.aclweb.org/anthology/J/J08/J08-2008.pdf

Moens (2006) Information extraction: algorithms and prospects in a retrieval context (just copy paste to Google -- no link allowed because of low reputation)

Sokolova & Lapalme (2009) A systematic analysis of performance measures for classification tasks http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Good answer upon a quick skim, but what is the difference? You define what they are, but not how they are different. $\endgroup$ – Behacad Mar 25 '14 at 6:50
  • $\begingroup$ @Behacad hello, thank you :) Look at the bottom it is explained. You cannot explain the difference between "somethings", before you don't explain what this "somethings" are. Nice day! $\endgroup$ – alex Mar 25 '14 at 8:59
  • $\begingroup$ At the bottom of what? Sorry I don't see it in text where the difference between them is explained. $\endgroup$ – Behacad Mar 25 '14 at 10:55
  • $\begingroup$ @Behacad Hello, If you look at the equations you see, that for the two class formulation and the general formulation, there are very distinctive comments. Then, at the end of the Accuracy there is a underlining "^^^^^^" where it is specifically mentioned why the equations are not equal. Wish you a nice day. $\endgroup$ – alex Mar 25 '14 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.