In my research I want to know how reliably certain feature of a sentence indicates the class that sentence belongs to.
So, according to that feature (=how many elements X they contain), the sentences have been classified in three groups A,B,C. E.g. Utterances have been classified as:
- A if they contain 1 X,
- B if they contain 2 Xs,
- and so on.
That classification was compared to a hand-coded gold-standard and I've built three confusion matrices (one for A sentences, one for B and one for C). E.g. for A:
- Hits: Sentence is A and contains 1 X.
- Misses: Sentence is A and contains 2 X
- False Alarms: Sentence is B and contains 1 X
- Correct Rejection: Sentence is B and contains 2 X
I was going to use d' to calculate how accurately the feature "Number of X" indicates sentence's class, but I see it has assumptions impossible to assess in yes/no tests (contrary to what happens in rating tests). Then I've read A' is not a non-parametric alternative as it was supposed to be (Macmillan & Creelman 1996).
What accuracy measure that takes all four possible outcomes (hits, misses, false alarms and correct rejections) into account and makes no distribution assumptions can I use?
I've also wanted to build an ROC plot (because I wanted to compare "Number of X to other features) but apparently it also assumes that the signal and noise distributions are normal and have equal variances. I've read some recent publications that talk about non-parametric ROC.. but I do not quite understand if they relax the normality and equal variance assumptions and if it is ok for me to use them..Help!
Edit Sometimes the prediction doesn't fit any of the classes the sentences belong to (= "Other"). Looking elsewhere (How to build ROC curve (or AUC) of classification model from confusion matrix only) I've found (very logically) that you can't build an ROC curve with just one data point (=confusion matrix). Although I have different classifiers that differ in some aspects they all share the same threshold, so I guess I shouldn't use it either. I'll go with classical classification performance measures.