In machine learning literature ROC curves are common performance measures. Quite recently published and not as popular (50+ citatations) are TOC curves as proposed by Pontius Jr, Robert Gilmore and Si, Kangping (2014).

  • Receiver operating characteristic (ROC) curve graphically illustrates TPR and FPR trade-offs as its threshold varies.

  • Total operating characteristic (TOC) depicts the underlying confusion matrix for each threshold. "TOC reveals more information than ROC"

A confusion matrix seems like an improper performance measure of a model, because its outcome depends on the (implicitly) specified decision rule/threshold.

What are reasons for and against TOC statistics in comparison to ROC or Precision-Recall curves? Also, shouldn't confusion matrices be replaced by TOC visualizations?

TOC curve example from Wikipedia: enter image description here

  • $\begingroup$ I can't think of a possibility to discuss this properly here. There are so many aspects that would need to be considered. Historically there are good reasons to use AUC as performance measure, and this doesn't depend on TOC or ROC. But then again this also depends... Maybe a real question would be easier to answer. $\endgroup$
    – cherub
    Mar 6, 2020 at 13:11
  • 1
    $\begingroup$ Yes I tried to be more specific in my edit. let's wait till that bounty ends and then this should be closed. Maybe someone comes up with a high level and or nice answer. $\endgroup$ Mar 6, 2020 at 15:57
  • 2
    $\begingroup$ I don't know about closing; I'd be interested in a nice discussion. But I fear that the format here is a bit too narrow. From my own point of view, I mainly used AUC in any case. For ROC curves, this is a seemingly natural argument (since it has a kind of obvious visibility) -- but less so for TOC. But this kind of "naturality" is always associated with its particular field of application. So even if TOC conveys more information, it is definitely more coded in the sense "harder to read". But notice that this is a completely different argument/question. $\endgroup$
    – cherub
    Mar 6, 2020 at 16:17
  • $\begingroup$ I've not encountered the TOC before, so thanks for raising the question. My first impression is that it does provide a comprehensive overview of the possibility space, but at the cost of more degrees of interpretation freedom. I'll have to look into it a bit more. $\endgroup$
    – ReneBt
    Mar 11, 2020 at 9:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.