I'm trying to design a continuous loss function for a logistic classifier.

Suppose I have the following confusion matrix:

[tn  fp
 fn  tp]

I want the loss function to be

A*tn + B*fn + C*fp + D*tp

where A, B, C, and D can be different. It's easy enough to implement this discrete version of course, but how can I make this cost function continuous so that my cost function is well-behaved when I try to minimize it?

  • 2
    $\begingroup$ If you wish to pursue your original goal of finding an extreme value for the stated discontinuous cost function, you might consult Google with the query "minimizing discontinuous functions". In particular, you might consult the Shor's book "Minimization methods for non-differentiable functions" and Pshnenichnyi's 2 books referenced therein. Also, Matlab has a function for numerical discontinuous minimization that may be of value. $\endgroup$ – George Kamin Sep 7 '16 at 0:03

I think you'll want to go one step back: look at the residuals of the predicted posterior probabilities instead of the discrete predicted class labels. Maybe go even further back and look at the residuals of the underlying linear model.

There's a bunch of literature on proper scoring rules that is of interest, e.g.
Gneiting, T. & Raftery, A. E. Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association, 102, 359-378 (2007). DOI: 10.1198/016214506000001437.

Also, you can combine the notion of continuous loss functions like mean squared errors with a differentiation of the different types and directions of the error.

Beleites, C. et al.: Validation of soft classification models using partial class memberships: An extended concept of sensitivity & Co. applied to grading of astrocytoma tissues, Chemom Intell Lab Syst, 122, 12 - 22 (2013). DOI: 10.1016/j.chemolab.2012.12.003
we discuss such continuous analoga of the usual classification error measures. While we derive these continuous measures for different reasons (uncertain reference) and without individual costs for each type of error, they also work for the usual classification setting with certain reference, and individual costs for the different "directions" of error can easily be added. The link goes to the project page of the implementation (R package).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.