What is the difference between accuracy and agreement?

Question

According to Manning et al. (p. 155) accuracy is the sum of the diagonal in the confusion matrix divided by the sum of all items. On the other hand, following Artstein and Poesio (p . 558) precisely this seems to be also called agreement (without chance correction).

Is there any conceptual difference?

Answer 1

EDITinfo: "accuracy" for multiclass was changed according to Moens (2006, p. 183)

EDIT: Sometimes (seemingly often) agreement and accuaracy are computed similarly. However, they seem to be used for different fields. Agreement is typically for inter-annotator tasks, while accuracy is the basic standard for classification (and other statistical tasks) performance assessment.

Agreement between coders in order to achieve reliability according to Artstein & Poesio (2008, p. 558) "the percentage of judgments on which the two analysts agree when coding the same data independently"

                       Coder 1
                 +-------------+
                  |   P  |  N   |                       
              +---+------+------+
              | P |  tp  |  fp  |                           
    Coder 2   +---+------+------+
              | N |  fn  |  tn  |                       
              +---+------+------+

SUM_all      = sum of all elements (sometimes called items)
NO_classes   = cardinality classes
SUM_diagonal = the sum of all elements on the main diagonal

TWO classes    Agreement =    (tp + tn) / SUM_all    (SIMILAR to accuracy)
i>2 classes    Agreement = SUM_diagonal / SUM_all    (NOT similar to accuracy)

EDIT: Accuracy of classification in order to measure the classification effectiveness (in context of classification denoted as performance -- qualitative) is computed at least two variants (1) Moens (2006, p. 183) accuracy AND (2) Sokolova & Lapalme (2008, p. 430) average accuracy)

                     Real Class
                  +-------------+
                  |   P  |  N   |                       
              +---+------+------+
 Hypothesized | P |  tp  |  fp  |                           
    Class     +---+------+------+
              | N |  fn  |  tn  |                       
              +---+------+------+


TWO classes             Accuracy =                      (tp + tn) / SUM_all     (SIMILAR to agreement)
i>2 classes (1)         Accuracy =                   SUM_diagonal / SUM_all     (according to Moens (2006, p. 183))
            (2)  averageAccuracy = SUM_i[(tp_i + tn_i) / SUM_all] / NO_classes  (not similar to Agreement or Accuracy (acc. to Sokolova, 2008, p. 430))
                                          ^^^^^^^^^^^^
                                          Note this are calculated relatively for each i
                                          => if the matrix is larger then 2x2 the tn are
                                             ALL other elements in the matrix
                                             which are  NOT associated with the i-class
                                             (i.e. from all other classes -- not only on the diagonal)!

Artstein & Poesio (2008) Inter-Coder Agreement for Computational Linguistics http://www.newdesign.aclweb.org/anthology/J/J08/J08-2008.pdf

Moens (2006) Information extraction: algorithms and prospects in a retrieval context (just copy paste to Google -- no link allowed because of low reputation)

Sokolova & Lapalme (2009) A systematic analysis of performance measures for classification tasks http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf