Are there standard evaluation procedures for non-binary classifiers?

In my case I have "nested" classes, being absence and presence of an effect the first and usual binary categorization, but further separating the positive class into to qualitative subclasses (1 and -1).

Overall this would be a ternary classification (0,-1,1), which makes two-dimensional evaluations unreliable (sensitivity, specificity and false discovery rates from binary true/false positives/negatives).

Any hints and ideas are appreciated.

  • 1
    $\begingroup$ I recommend to start reading on structured prediction, which focuses on building models that predict structured objects instead of scalar values. In your case, the output structure can be modelled as a graph. Essentially, you will end up crafting a custom cost function that defines the trade-offs in your problem (i.e., costs of false positives and false negatives of each of your target classes). $\endgroup$ Jun 13 '16 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.