Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a very simple data set consisting of three columns: Ground Truth Canopy Class, Method 1 Canopy Class and Method 2 Canopy Class. Each row in the columns represents the canopy class (i.e. 1 through 5). I have produced an error matrix in excel and calculated the overall accuracy and Khat (Figure 1). However, now I need to test whether or not method one is significantly different from method two using the attached equations (Figure 2 and 3). I could use help to calculate the following using R or Excel:

  1. Z-score
  2. K hat
  3. Variance of K hat

Is there a package in R I've overlooked that can aid me these calculations?

Figure 1

enter image description here

Figure 2

Figure 3

Source: Assessing the Accuracy of Remotely Sensed Data (Congalton, 2009)

share|improve this question
If you predicted canopy class with both methods for each sample, your have a paired design. I.e. your confusion tables are not independent. – cbeleites Nov 22 '12 at 10:56
Your canopy seems to be a continuous outcome (fraction of canopy coverage) which is then cut into ordered classes. In that case, why not do regression, and use regression error measures? – cbeleites Nov 22 '12 at 10:59
@cbeleites I have already done regression based analysis using the equivalence package in R. I believe both of my error matrices are independent--I provided an example of an error matrix between max likelihood classification and reference data. I also have another error matrix that is not displayed which is between random forest classification and reference data. – Aaron Nov 22 '12 at 20:39
wrt independence: you say that you have 3 columns: reference, pred 1, pred 2. So you have predictions from both classifiers for the same sample, right? – cbeleites Nov 22 '12 at 21:10
error matrix 1 is based off pred 1 x reference; error matrix 2 is based off pred 2 x reference. I need the error matrix to describe omission and commission error rates. – Aaron Nov 22 '12 at 21:25
up vote 1 down vote accepted

It might not seem that way off the bat, but (assuming, as you said, independence between the two methods) your question fits nicely into the framework of a survey with multiple raters. Check out the survey package, specifically the svydesign, svycontrast, and svykappa functions. HTH,

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.