Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm trying to design a continuous loss function for a logistic classifier.
Suppose I have the following confusion matrix:
[tn fp
fn tp]
I want the loss function to be
A*tn + B*fn + C*fp + D*tp
where A, B, C, and D can be different. It's easy enough to implement this discrete version of course, but how can I make this cost function continuous so that my cost function is well-behaved when I try to minimize it?
I think you'll want to go one step back: look at the residuals of the predicted posterior probabilities instead of the discrete predicted class labels. Maybe go even further back and look at the residuals of the underlying linear model.
There's a bunch of literature on proper scoring rules that is of interest, e.g.
Gneiting, T. & Raftery, A. E. Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association, 102, 359-378 (2007). DOI: 10.1198/016214506000001437.
Also, you can combine the notion of continuous loss functions like mean squared errors with a differentiation of the different types and directions of the error.
In
Beleites, C. et al.: Validation of soft classification models using partial class memberships: An extended concept of sensitivity & Co. applied to grading of astrocytoma tissues, Chemom Intell Lab Syst, 122, 12 - 22 (2013). DOI: 10.1016/j.chemolab.2012.12.003
we discuss such continuous analoga of the usual classification error measures. While we derive these continuous measures for different reasons (uncertain reference) and without individual costs for each type of error, they also work for the usual classification setting with certain reference, and individual costs for the different "directions" of error can easily be added. The link goes to the project page of the implementation (R package).
