How should I calculate performance with 3 kinds of response (yes, no and not applicable)? I am having trouble calculating employee performance.
I have an observation form with 34 behaviours, but not all of the behaviours are observed at one given time because it depends on the topic being discussed. There are 3 options to mark on the observation form: yes (it means the behaviour was observed), no (behaviour wasn't observed) and NA (behaviour was not applicable because the topic did not come up). There are 2 observations per month for each employee, some behaviours come up once per month, some once in 3 months, therefore I need to input all the data yes, no and NA. How can I calculate performance in this situation?  
Is it correct to use Item Response Theory? 
(Mentions of R packages in which the analysis could be done would also be helpful)
 A: A latent trait approach with IRT should work fine here. The idea is to find out what each persons expected total score would have been if they had no missing data and use that for comparisons instead of the observed total score (after removing NAs). This allows individuals to be compared more easily by transforming the scores onto the same metric while accounting for the variability due to the missing data. This is generally better than basing information on their raw total scores (which will be inversely correlated with the number of missing elements) because the NAs only serve to decrease the precision in the estimates instead of affecting the expected values.  
First obtain $\hat{\theta}$ estimates for each individual and then transform these back into the metric based on the expected total score. 
Here's an example in R:
library(mirt)

set.seed(1)
dat <- simdata(matrix(1, 34), matrix(rnorm(34)), 100, itemtype = 'dich')
dat[sample(1:length(dat), 200)] <- NA

mod <- mirt(dat, 1, itemtype = 'Rasch')
fs <- fscores(mod, full.scores.SE = TRUE)

#transform back into meaningful metric
upper <- fs[,1] + 1.96 * fs[,2]
lower <- fs[,1] - 1.96 * fs[,2]
T <- expected.test(mod, fs[,1, drop=FALSE])
T.u <- expected.test(mod, matrix(upper))
T.l <- expected.test(mod, matrix(lower))
df <- data.frame(T, T.l, T.u)
head(df)
         T       T.l      T.u
1 14.20269  9.691575 19.03195
2 13.11440  8.656549 18.07226
3 15.83404 10.949113 20.80466
4 18.91060 14.017607 23.43629
5 22.80185 17.976009 26.74832
6 25.92742 21.368552 29.24223

The T in the example is the expected total score for the individual (what they would be expected to get had the other behaviours been observed and not missing) while the last two columns represent the uncertainty in this score. You may find it useful to use some itemtype other than 'Rasch' though if some behaviours are better reflective of performance than others (e.g. '2PL'). This example assumes the indicators are all of equal importance.  
