# What is the implementation of hinge loss in Tensorflow?

$L_i$ is the hinge loss here. $y_i$ the correct class. Rest is self explanatory

from wikipedia i get:

Weston and Watkins: $L_i = \sum_{j\neq y_i} \max(0, w_j^T x_i - w_{y_i}^T x_i + 1)$

Crammer and Singer: $L_i = \max(0, max_{j\neq y_i}(w_j^T x_i) - w_{y_i}^T x_i + 1)$

and here is the implementation of hinge loss from tensorflow:

if labels is None:
raise ValueError("labels must not be None.")

if logits is None:
raise ValueError("logits must not be None.")
with ops.name_scope(scope, "hinge_loss", (logits, labels, weights)) as scope:
logits = math_ops.to_float(logits)
labels = math_ops.to_float(labels)
logits.get_shape().assert_is_compatible_with(labels.get_shape())
# We first need to convert binary labels to -1/1 labels (as floats).
all_ones = array_ops.ones_like(labels)
labels = math_ops.subtract(2 * labels, all_ones)
losses = nn_ops.relu(
math_ops.subtract(all_ones, math_ops.multiply(labels, logits)))
return compute_weighted_loss(
losses, weights, scope, loss_collection, reduction=reduction)


I am not able to reconcile this implementation to any of the definitions given above. What is the implementation of hinge loss in the Tensorflow?

Please note that compute_weighted_loss is just the weighted average of all the elements. weights is a parameter to the functions which is generally, and at default, a tensor of all ones

• Could you add hyperlink references to the Weston-Clarke and Crammer-Singer definitions? Thank you.
– Jim
Jun 13, 2018 at 20:32

$\ell(y) = \text{max}(0, 1 - t \cdot y)$
This assumes your labels are in a $\pm1$ binary, per the TensorFlow code you linked to and the Wiki page. The correspondence between the equation above and the code is:
• $t$ is labels = math_ops.subtract(2 * labels, all_ones), which assumes labels comes in as a tensor with values of $[0,1]$
• $y$ is the direct output of your model (which is why they're referred to as the logits)
• $1 - t \cdot y$ is captured by math_ops.subtract(all_ones, math_ops.multiply(labels, logits))