3
$\begingroup$

The triplet loss is defined by Florian Schroff, Dmitry Kalenichenko, James Philbin in "FaceNet: A Unified Embedding for Face Recognition and Clustering" as $$ \mathcal L = \sum_\mathcal T \max\left\{0, \left\|f(x^a) - f(x^p) \right\|_2^2 - \left\|f(x^a)-f(x^n)\right\|_2^2 +\alpha\right\} $$ where $\mathcal T$ is a set of triplets $x^a, x^p, x^n$.

The loss measures the squared Euclidean distance between three points $x^a, x^p, x^n$. The goal is to train an embedding $f$ of the data $x$ so that the members of the same class (positives) are close together (in the sense that $\left\|f(x^a_i) - f(x^p_i) \right\|_2^2$ is small) and also far apart (in the sense that $\left\|f(x_i^a)-f(x_i^n)\right\|_2^2$ is large) from every other class (negatives). We desire the classes to be separated by a margin $\alpha$.

The authors write "We constrain this embedding to live on the $d$-dimensional hypersphere, i.e. $\| f(x) \|_2 = 1$."

Taken together, these two facts strike me as odd. If the points are on the surface of a (hyper)sphere, then the Euclidean distance computation is measuring the chord distance -- the distance between the two points cuts through the sphere. This will always be shorter than the distance measured over the surface of the sphere.

Alternatively, instead of measuring the chord distance, we could measure the distance over the surface of the (hyper)sphere. The geodesic distance is proportional to the angle formed by the two points; on the unit sphere, the distance is: $$ d_G = \cos^{-1}\left( f(x_i) \cdot f(x_j) \right). $$ The distance $d_G$ is in $[0,\pi]$. (Remember, we've already normalized the $\|f(x)\|_2=1$.)

The difference seems crucial to me. Minimizing the Euclidean distance would suggest that you move a point through the sphere -- but this is impossible, due to the constraint that all of the points must be located on the surface. By contrast, minimizing the spherical distance just moves points around on the sphere's surface. (Of course, they address this impossibility by re-normalizing everything to have unit length after applying each update, but one must wonder if this practice is papering over the real problem, which is that the gradients don't point in the right direction.)

Importantly, this does not appear to be a setting where fudging huge distances is okay.

Large distances between the $x$s arise in online negative mining, which only compute the loss for the anchor-negative pairs for which the loss is the highest. (This speeds up training.) This situation is the exact scenario suggested in the Face Net paper.

It's easy to construct $x$s that have gradients that point in different directions for each of the two distances, so it seems highly consequential for gradient-based methods of model training (i.e. back-propagation).

In the paper, the authors write that they used different negative mining strategies to prevent model collapse. It's natural to wonder whether the risk of model collapse is exacerbated by the choice of Euclidean distance for the loss computation, and that this could be ameliorated by using geodesic distances.

The FaceNet authors write about the great pains they took to avoid model collapse. Would switching to a geodesic distance prevent, or at least mitigate, model collapse? Or, asked slightly differently, how can one demonstrate that the choice of squared Euclidean distance is or isn't inhibiting learning for triplet models?

$\endgroup$
13
  • $\begingroup$ A topologist would remind us that all metrics are equivalent in finite-dimensional spaces... seriously, perhaps it simply makes little difference? If points on the sphere are close together in either metric, they are also close together in the other metric. If they are far apart (e.g., antipodal), then yes, the difference between the two metrics is larger - but maybe this is simply captured when we optimize $\alpha$? $\endgroup$ Commented Sep 19, 2023 at 14:32
  • 1
    $\begingroup$ @Dave: no, it's definitely not the same. MSE and RMSE have the same minimizer, since the square root it monotonic. Here, we do get different minimizers. What I'm thinking is that the difference between the minimizers are possibly so small (and more importantly, the difference in holdout performance between the minimizers is so small) that a tradeoff towards more comprehensibility in the code makes sense. (But that of course is only a hypothesis of mine.) $\endgroup$ Commented Sep 19, 2023 at 15:59
  • 1
    $\begingroup$ @Stephan Your topologist would be incorrect, but the underlying idea is right: the chordal distance and the geodesic distance are metrically equivalent on the sphere. Not only that, they are identical in infinitesimal neighborhoods. Thus, in applications where getting small distances correct and fudging the huge distances is okay, people use the computationally and analytically simpler Euclidean distance. $\endgroup$
    – whuber
    Commented Sep 19, 2023 at 16:23
  • 2
    $\begingroup$ When a topologist says equivalent they mean topologically equivalent, which just means that the open sets are the same. we shouldn't conclude from this that they are practically equivalent; otherwise, Ridge and Lasso would be the same thing. $\endgroup$ Commented Sep 19, 2023 at 17:08
  • 1
    $\begingroup$ Sorry for starting my comment with a factually false and misleading throwaway observation, which seems to be derailing the entire conversation... Does anyone want to go back to the original question? I guess the best course of action would be to email one of the original autors; at least Schroff seems to be still active. If you do and get an answer, please post here... Then again, if you asked me for the rationale of a decision in a paper I wrote eight years ago, I might be a bit stumped... $\endgroup$ Commented Sep 19, 2023 at 19:04

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.