The triplet loss is defined by Florian Schroff, Dmitry Kalenichenko, James Philbin in "FaceNet: A Unified Embedding for Face Recognition and Clustering" as $$ \mathcal L = \sum_\mathcal T \max\left\{0, \left\|f(x^a) - f(x^p) \right\|_2^2 - \left\|f(x^a)-f(x^n)\right\|_2^2 +\alpha\right\} $$ where $\mathcal T$ is a set of triplets $x^a, x^p, x^n$.
The loss measures the squared Euclidean distance between three points $x^a, x^p, x^n$. The goal is to train an embedding $f$ of the data $x$ so that the members of the same class (positives) are close together (in the sense that $\left\|f(x^a_i) - f(x^p_i) \right\|_2^2$ is small) and also far apart (in the sense that $\left\|f(x_i^a)-f(x_i^n)\right\|_2^2$ is large) from every other class (negatives). We desire the classes to be separated by a margin $\alpha$.
The authors write "We constrain this embedding to live on the $d$-dimensional hypersphere, i.e. $\| f(x) \|_2 = 1$."
Taken together, these two facts strike me as odd. If the points are on the surface of a (hyper)sphere, then the Euclidean distance computation is measuring the chord distance -- the distance between the two points cuts through the sphere. This will always be shorter than the distance measured over the surface of the sphere.
Alternatively, instead of measuring the chord distance, we could measure the distance over the surface of the (hyper)sphere. The geodesic distance is proportional to the angle formed by the two points; on the unit sphere, the distance is: $$ d_G = \cos^{-1}\left( f(x_i) \cdot f(x_j) \right). $$ The distance $d_G$ is in $[0,\pi]$. (Remember, we've already normalized the $\|f(x)\|_2=1$.)
The difference seems crucial to me. Minimizing the Euclidean distance would suggest that you move a point through the sphere -- but this is impossible, due to the constraint that all of the points must be located on the surface. By contrast, minimizing the spherical distance just moves points around on the sphere's surface. (Of course, they address this impossibility by re-normalizing everything to have unit length after applying each update, but one must wonder if this practice is papering over the real problem, which is that the gradients don't point in the right direction.)
Importantly, this does not appear to be a setting where fudging huge distances is okay.
Large distances between the $x$s arise in online negative mining, which only compute the loss for the anchor-negative pairs for which the loss is the highest. (This speeds up training.) This situation is the exact scenario suggested in the Face Net paper.
It's easy to construct $x$s that have gradients that point in different directions for each of the two distances, so it seems highly consequential for gradient-based methods of model training (i.e. back-propagation).
In the paper, the authors write that they used different negative mining strategies to prevent model collapse. It's natural to wonder whether the risk of model collapse is exacerbated by the choice of Euclidean distance for the loss computation, and that this could be ameliorated by using geodesic distances.
The FaceNet authors write about the great pains they took to avoid model collapse. Would switching to a geodesic distance prevent, or at least mitigate, model collapse? Or, asked slightly differently, how can one demonstrate that the choice of squared Euclidean distance is or isn't inhibiting learning for triplet models?
