1
$\begingroup$

This question is about comparing the significant difference between accuracy metrics (that can be derived from a confusion matrix) calculated for four different models.

I have four unsupervised change detection models that I want to compare in terms of accuracy on a large dataset ($n$ > 3k). For every sampling location, each model will either detect 'change' or 'no change'. I am planning to run them all on the same dataset, and compare the accuracy results for a given metric. Since I suspect the four models will perform very similarly on any type of accuracy metric, I want to calculate the significant difference between the accuracy results of the 4 models.

I read that I could calculate the Cohen's Kappa Coefficient where I use a model prediction as one 'rater' and the validation data set as the second 'rater'. This would give the agreement between the model and reality. With the z-score to compare two kappa scores relative to each other, so whether the agreement levels between model A and reality are significantly different than the agreement between model B and reality. It would look like this:

equation

where the variances in the denominator can be calculated using the equation in this article (first page).

Now, my problem is: the z-score on kappa values only works pair-wise. To assess all significant differences between kappa coefficients of the 4 models, I would have to run the z-score 6 times. Besides this being inefficient, it may also lead to multiple testing issues, especially if I want to break the analysis down into different variables present in the dataset later on.

Would there be a way of testing the 4 accuracy values (like the Cohen's Kappa for instance) at the same time?

$\endgroup$
1
  • $\begingroup$ I think that formula for $z$ is appropriate for independent $\kappa$ but yours are based on the same data-set if I understand you correctly. $\endgroup$
    – mdewey
    Feb 16 at 11:10
1
$\begingroup$

Those cases in which all your four methods give the same result, whether that is correct or not, are uninformative for assessing the differential performance of the four methods, so they can be excluded.

Now split your dataset by the true response: change or no change.

For each of these, just form the four by two table of method by correct and use any test of your choice on the resulting contingency table. If you are not interested in whether the methods have different performance in each split, you can combine them. You could also form a $2 \times{} 2 \times{} 4$ table and fit a log-linear model to it.

So, in short, I am not convinced that $\kappa$ is the answer here.

$\endgroup$
0
$\begingroup$

Fleiss' kappa is a multi-rater generalization of Scott's pi.

The generalization of Cohen's kappa is described in

Davies, Mark, and Joseph L. Fleiss. "Measuring agreement for multinomial data." Biometrics (1982): 1047-1051.

Since you only have two responses "change" or "no change", it is relatively easy to calculate the estimate and the standard error for your situation using the formulas in the paper.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your answer! From what I found about Fleiss' Kappa, is that it seems to measure whether the agreement between (in this case) the 4 models is more than chance. I think that when I would use it this way, the validity of the votes cast by individual models is not taken into account. So for example the 4 models could all say "it changed" in London, whereas my validation data could hold "no change" in London. The inter-model agreement would be very high in this case, and maybe above chance levels, but it doesn't indicate whether the vote (change) was the right choice. $\endgroup$
    – saQuist
    Feb 16 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.