Comprehensive overview of loss functions?

Question

I am trying to get a global perspective on some of the essential ideas in machine learning, and I was wondering if there is a comprehensive treatment of the different notions of loss (squared, log, hinge, proxy, etc.). I was thinking something along the lines of a more comprehensive, formal presentation of John Langford’s excellent post on Loss Function Semantics.

Answer 1

The Tutorial on Energy-Based Learning by LeCun et al. might get you a good part of the way there. They describe a number of loss functions and discuss what makes them "good or bad" for energy based models.

Answer 2

Well, there's this and that. Two papers by Cramer and others discussing loss in the context of online learning algorithms.

Answer 3

The loss function is given by the problem. It could be anything. For example, you could also penalize the used CPU time and space.

In reinforcement learning, the loss function is an unknown non-deterministic function. You cannot redefine it without changing the problem.

Answer 4

I know this question is sort of dated, but it is something I am currently interested in.

A very good paper on the topic of convex loss functions and classifier consistency is "Statistical Behavior and Consistency of Classification Methods Based on Convex Risk Minimization" by Tong Zhang.