Appropriate statistic for testing similarity of two paired binary datasets (of glaciovolcanism) I am studying the ice-covered volcanoes of the world. I have developed three methods that use existing datasets to determine whether a particular volcano is icy or not, and I applied all three methods (PZL, RGL, MDL) to a database of 1444 volcanoes. So, I have three binary vectors of length 1444.
I would like to know to what extent these methods are in agreement with each other about whether each volcano is ice-covered or not. McNemar's test seemed appropriate for this at first so I tried it and my p-values for rejecting the null hypothesis (of similarity) were:
    PZL         RGL         MDL
PZL 1.00E+000   1.51E-021   1.52E-037
RGL 1.51E-021   1.00E+000   2.66E-005
MDL 1.52E-037   2.66E-005   1.00E+000

This is not particularly useful. Both method 1 and method 2 detect about 200 icy volcanoes volcanoes each (out of 1444 volcanoes), and the majority (~170) are classified as icy by both, so intuitively it seems like these two vectors are fairly similar.
Can someone recommend a test of similarity between these methods which is more in line with my intuition? Essentially I'm interested in the Hamming distance between the pairs so maybe I just need to compare the Hamming distance to the expected Hamming distance or something?
 A: Your question is "to what extent these methods are in agreement".  That means you need to use statistical methods that test for agreement.  Because these data are categorical (binary), kappa is in order; since you have three methods, Fleiss' kappa seems the obvious choice.  
I would start with Fleiss' kappa.  This will let you test if there is more agreement than you would expect by chance alone.  (You presumably want to find a significant result, indicating that they do agree.)  Following that, I might try three pairwise tests using Cohen's kappa.  
Those provide tests of agreement.  You may also want to know the magnitude of agreement, and the nature of their disagreement.  Many people take the kappa statistic as a measure of effect size.  John Uebersax has argued that we should not think of the 'chance-corrected' measure, kappa, as an appropriate measure of effect size.  Instead, you can simply describe the level of raw agreement.  
Given a lack of perfect agreement, you may want to describe how they disagree.  We can loosely think of there being two components of the lack of perfect agreement: bias and unreliability.  Bias would imply that one method is systematically more likely to say 'icy' than the other.  That is what McNemar's test is telling you.  It is fine to use McNemar's test in this way, so long as you understand that is what you are doing.  You can also descriptively state the marginal proportions.  (Unreliability here probably doesn't need to be further quantified; it is just the lack of agreement beyond the bias.)  If you have additional information on individual volcanoes, you could explore how that is related to cases where one method says 'yes' and another says 'no'.  (For example, one method might rely more on measures of reflectivity in satellite images, while another weights temperature measures more heavily.)  
