$L_i$ is the hinge loss here. $y_i$ the correct class. Rest is self explanatory

from wikipedia i get:

Weston and Watkins: $L_i = \sum_{j\neq y_i} \max(0, w_j^T x_i - w_{y_i}^T x_i + 1)$

Crammer and Singer: $L_i = \max(0, max_{j\neq y_i}(w_j^T x_i) - w_{y_i}^T x_i + 1)$

and here is the implementation of hinge loss from tensorflow:

if labels is None:
    raise ValueError("labels must not be None.")

if logits is None:
    raise ValueError("logits must not be None.")
with ops.name_scope(scope, "hinge_loss", (logits, labels, weights)) as scope:
    logits = math_ops.to_float(logits)
    labels = math_ops.to_float(labels)
    # We first need to convert binary labels to -1/1 labels (as floats).
    all_ones = array_ops.ones_like(labels)
    labels = math_ops.subtract(2 * labels, all_ones)
    losses = nn_ops.relu(
        math_ops.subtract(all_ones, math_ops.multiply(labels, logits)))
    return compute_weighted_loss(
        losses, weights, scope, loss_collection, reduction=reduction)

I am not able to reconcile this implementation to any of the definitions given above. What is the implementation of hinge loss in the Tensorflow?

Please note that compute_weighted_loss is just the weighted average of all the elements. weights is a parameter to the functions which is generally, and at default, a tensor of all ones

  • $\begingroup$ Could you add hyperlink references to the Weston-Clarke and Crammer-Singer definitions? Thank you. $\endgroup$
    – Jim
    Jun 13, 2018 at 20:32
  • $\begingroup$ made the link available @Jim $\endgroup$ Jun 13, 2018 at 21:35

1 Answer 1


It looks like the very first version of hinge loss on the Wikipedia page.

That first version, for reference:

$\ell(y) = \text{max}(0, 1 - t \cdot y)$

This assumes your labels are in a $\pm1$ binary, per the TensorFlow code you linked to and the Wiki page. The correspondence between the equation above and the code is:

  • $t$ is labels = math_ops.subtract(2 * labels, all_ones), which assumes labels comes in as a tensor with values of $[0,1]$
  • $y$ is the direct output of your model (which is why they're referred to as the logits)
  • $1 - t \cdot y$ is captured by math_ops.subtract(all_ones, math_ops.multiply(labels, logits))
  • 1
    $\begingroup$ please refer to the specific multinomial extension i have linked to in the recent edit in question. what you are referring to is a binary classification problem formulation $\endgroup$ Jun 13, 2018 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.