EDITinfo: "accuracy" for multiclass was changed according to Moens (2006, p. 183)
EDIT: Sometimes (seemingly often) agreement and accuaracy are computed similarly. However, they seem to be used for different fields. Agreement is typically for inter-annotator tasks, while accuracy is the basic standard for classification (and other statistical tasks) performance assessment.
Agreement between coders in order to achieve reliability according to Artstein & Poesio (2008, p. 558) "the percentage of judgments on which the two analysts agree when coding the same data independently"
Coder 1
+-------------+
| P | N |
+---+------+------+
| P | tp | fp |
Coder 2 +---+------+------+
| N | fn | tn |
+---+------+------+
SUM_all = sum of all elements (sometimes called items)
NO_classes = cardinality classes
SUM_diagonal = the sum of all elements on the main diagonal
TWO classes Agreement = (tp + tn) / SUM_all (SIMILAR to accuracy)
i>2 classes Agreement = SUM_diagonal / SUM_all (NOT similar to accuracy)
EDIT: Accuracy of classification in order to measure the classification effectiveness (in context of classification denoted as performance -- qualitative) is computed at least two variants (1) Moens (2006, p. 183) accuracy AND (2) Sokolova & Lapalme (2008, p. 430) average accuracy)
Real Class
+-------------+
| P | N |
+---+------+------+
Hypothesized | P | tp | fp |
Class +---+------+------+
| N | fn | tn |
+---+------+------+
TWO classes Accuracy = (tp + tn) / SUM_all (SIMILAR to agreement)
i>2 classes (1) Accuracy = SUM_diagonal / SUM_all (according to Moens (2006, p. 183))
(2) averageAccuracy = SUM_i[(tp_i + tn_i) / SUM_all] / NO_classes (not similar to Agreement or Accuracy (acc. to Sokolova, 2008, p. 430))
^^^^^^^^^^^^
Note this are calculated relatively for each i
=> if the matrix is larger then 2x2 the tn are
ALL other elements in the matrix
which are NOT associated with the i-class
(i.e. from all other classes -- not only on the diagonal)!
Artstein & Poesio (2008) Inter-Coder Agreement for Computational Linguistics
http://www.newdesign.aclweb.org/anthology/J/J08/J08-2008.pdf
Moens (2006) Information extraction: algorithms and prospects in a retrieval context (just copy paste to Google -- no link allowed because of low reputation)
Sokolova & Lapalme (2009) A systematic analysis of performance measures for classification tasks
http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf