# How to calculate error matrices, K hat and var(K hat)?

I have a very simple data set consisting of three columns: Ground Truth Canopy Class, Method 1 Canopy Class and Method 2 Canopy Class. Each row in the columns represents the canopy class (i.e. 1 through 5). I have produced an error matrix in excel and calculated the overall accuracy and Khat (Figure 1). However, now I need to test whether or not method one is significantly different from method two using the attached equations (Figure 2 and 3). I could use help to calculate the following using R or Excel:

1. Z-score
2. K hat
3. Variance of K hat

Is there a package in R I've overlooked that can aid me these calculations?

Figure 1

Figure 2

Figure 3

Source: Assessing the Accuracy of Remotely Sensed Data (Congalton, 2009)

-
If you predicted canopy class with both methods for each sample, your have a paired design. I.e. your confusion tables are not independent. – cbeleites Nov 22 '12 at 10:56
Your canopy seems to be a continuous outcome (fraction of canopy coverage) which is then cut into ordered classes. In that case, why not do regression, and use regression error measures? – cbeleites Nov 22 '12 at 10:59
@cbeleites I have already done regression based analysis using the equivalence package in R. I believe both of my error matrices are independent--I provided an example of an error matrix between max likelihood classification and reference data. I also have another error matrix that is not displayed which is between random forest classification and reference data. – Aaron Nov 22 '12 at 20:39
wrt independence: you say that you have 3 columns: reference, pred 1, pred 2. So you have predictions from both classifiers for the same sample, right? – cbeleites Nov 22 '12 at 21:10
error matrix 1 is based off pred 1 x reference; error matrix 2 is based off pred 2 x reference. I need the error matrix to describe omission and commission error rates. – Aaron Nov 22 '12 at 21:25

It might not seem that way off the bat, but (assuming, as you said, independence between the two methods) your question fits nicely into the framework of a survey with multiple raters. Check out the survey package, specifically the svydesign, svycontrast, and svykappa functions. HTH,