Understand the idea of margin in contrastive loss for siamese networks

Question

I was studying siamese networks for authentication. Loss is:

Y is 0 for dissimilar pairs and 1 for similar pairs.

D_w is the distance (e.g. euclidean distance) between two pairs (by using weights w).

If pairs are similar, then loss is equal to the green box in loss function. If pairs are dissimilar, then loss is equal to the red box in the loss function.

Suppose that we have two similar samples, then the loss is equal to the distance^2 between them. Since the aim of the training is to minimize the loss, therefore by training on similar pairs, will help to minimize the distance between similar pairs.

If we have two dissimilar samples, therefore there is an m term, which signifies margin in the loss function. According to the red box, if margin<D_w, then, loss is equal to zero. What exactly signifies the margin. What we are separating with margin? Suppose that I want to recognize face x (aim: to understand input is face x or not), what margin does? Is it something linear (like SVM margin), circle or another form? Should it have a specific form? Why if margin<D_w, loss does nothing. I do not understand this part. I do not understand if it is a learnable parameter or a pre-defined value? If it is a pre-defined value, how do we predefine it, based on which knowledge?

Edit 1: I found this explanation but I did not understand. I put it, in case it helps you to understand: taken from here:

Role of margin m: It is to be noted that the representations of negative pairs will only contribute to the loss if the estimated distance ||f(x)-f(y)|| < m. Meaning that it will no longer care how far the negative pairs x and y are once this limit reaches. So, it can focus more on the difficult to embed points.

Answer 1

If you look at the Loss it is composed of two parts, 1st the loss when they are similar and second the loss when they are dissimilar.

1. If model works very well, observation which are similar should have very small distance. Using Sqaure distance in first ensure that model is penalized if its gives high distance for similar observation (classes)

2. For Dissimilar observation, if two class are seprated well enough defined by Margin then the error contirbution is zero. Lets look at the example below :

   Suppose we choose margin to be 10, and we have two observation where distance between them is 19,1 lets see how loss looks like

       Error Observation 1 = max(0,10-19)^2   = 0  - No Contribution to error
       Error Observation 2 = max(0,10-1)^2    = 81  - High Contribution to error
   

As objective is to minimize loss, Observation 2 which is difficult to embed would be contibuting to loss so model can optimise for that

So margin ensure observation which have was well separated i.e. Distance Greater than Margin their contribution to error is zero. So optimisation algorithm can concentrate on seprating difficult Data Point in above example Observation 2. Thus Margin Helps on optimisation to embed difficult to seprate points

Answer 2

The best explanation is given in A Tutorial on Energy-Based Learning by LeCun et al, concretely in section 5. Also, Learning a Similarity Metric Discriminatively, with Application to Face Verification by Chopra et al. provides a detailed analysis for the case of face verification. The motivation for introducing the margin is to avoid a collapsed solution (where the resulting energy landscape is constant or zero, that is, the model is not able to distinguish between good and bad solutions).

Training will be successful when the model associates lower energy values to correct answers, and higher energy values to incorrect answers. The paper describes in detail, what properties the loss function must fulfill. If you write the above expression in a slightly more abstract manner: $YL_G + (1-Y)L_I$ where $L_G$ is the loss for the similar pair of samples, and $L_I$ for the dissimilar one, how should these loss functions behave?.

You want $L_G$ to be monotonically increasing, so that the network learns to pull down the energy values for similar samples, and $L_I$ monotonically decreasing to do the opposite for dissimilar samples. The contrastive loss is just one such loss function that fulfills that condition.

Now, why the margin? If $m=0$ notice that both terms would then match, and after training you would be giving all samples the same energy value. This effect is shown in the following figures,

After introducing the margin, you have a well-behaved loss function

Both figures are taken from the first reference.

Answer 3

In addition to the wonderful explanations given above, the idea of the "margin" affects only the dissimilar case. We want two (truly) dissimilar parts to be dissimilar, but by how much? The "margin" gives you a threshold where dissimilarity error doesn't matter anymore because at such a threshold they are clearly far apart and making them more dissimilar won't help the algorithm. Rather, you want to focus on hard examples e.g. dissimilar parts that fall within the margin (appearing as though they are similar), as seen in observation 2 in @Ashwiniku918's post.