1
$\begingroup$

In my research I want to know how reliably certain feature of a sentence indicates the class that sentence belongs to.

So, according to that feature (=how many elements X they contain), the sentences have been classified in three groups A,B,C. E.g. Utterances have been classified as:

  • A if they contain 1 X,
  • B if they contain 2 Xs,
  • and so on.

That classification was compared to a hand-coded gold-standard and I've built three confusion matrices (one for A sentences, one for B and one for C). E.g. for A:

  • Hits: Sentence is A and contains 1 X.
  • Misses: Sentence is A and contains 2 X
  • False Alarms: Sentence is B and contains 1 X
  • Correct Rejection: Sentence is B and contains 2 X enter image description here

I was going to use d' to calculate how accurately the feature "Number of X" indicates sentence's class, but I see it has assumptions impossible to assess in yes/no tests (contrary to what happens in rating tests). Then I've read A' is not a non-parametric alternative as it was supposed to be (Macmillan & Creelman 1996).

What accuracy measure that takes all four possible outcomes (hits, misses, false alarms and correct rejections) into account and makes no distribution assumptions can I use?

I've also wanted to build an ROC plot (because I wanted to compare "Number of X to other features) but apparently it also assumes that the signal and noise distributions are normal and have equal variances. I've read some recent publications that talk about non-parametric ROC.. but I do not quite understand if they relax the normality and equal variance assumptions and if it is ok for me to use them..Help!

Edit Sometimes the prediction doesn't fit any of the classes the sentences belong to (= "Other"). Looking elsewhere (How to build ROC curve (or AUC) of classification model from confusion matrix only) I've found (very logically) that you can't build an ROC curve with just one data point (=confusion matrix). Although I have different classifiers that differ in some aspects they all share the same threshold, so I guess I shouldn't use it either. I'll go with classical classification performance measures.

Improve this question
$\endgroup$
4
  • $\begingroup$ Why do you need a signal-detection-based measure? Why not use any old classification measure? Why do you need your measure to be free from distributional assumptions? Can you paste in the confusion matrix? Are the classes balanced? $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 31, 2020 at 1:19
  • $\begingroup$ I was suggested to use a signal detection measure. Previously, I've been using a hit/miss rate which left outcomes outside. $\endgroup$
    – Leandra
    Commented Jan 31, 2020 at 3:42
  • 2
    $\begingroup$ Too short for an answer, but: the ROC curve does not make assumptions about the distribution of the data. $\endgroup$
    – Calimo
    Commented Jan 31, 2020 at 6:43
  • $\begingroup$ In Stanislaw & Todorov (1999) it says: "The ROC area can be calculated from yes/no data, as well as from rating data. One method involves using d' to estimate the ROC area; the resulting measure is called Ad" This measure is valid only when the d' assumptions are met. A' also estimates the ROC area, and does so without assuming that the decision variable has a particular (e.g., normal) distribution. However, A' is problematic in other respects (Macmillan & Kaplan, 1985; W D. Smith, 1995)." $\endgroup$
    – Leandra
    Commented Jan 31, 2020 at 14:10

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.