I'm implementing a research paper on GANs and have come across this rather convoluted text-image loss function which has these main components:

$$P(D_i | Q_i) = \frac{\exp({\gamma_3 R(Q_i, D_i)})}{\sum^N_{j=1}\exp(\gamma_3R(Q_i, D_j))}$$

  • Where $D$ is a text tensor of shape $(N, T, D)$ and $Q$ is an image tensor of shape $(N, 289, D)$. In this case, $N$ refers to the batch size, $T$ refers to the length of the text encoding, and $D$ refers to the dimensions of the word vector encoded in the embedding layer. Intuitively, you may see $D$ as a batch of encoded text descriptions, while $Q$ as a batch of encoded image features.

  • $R(Q_i, D_i)$ measures the cosine similarity score between some image-text pair of $Q_i$ and $D_i$ via another formula, which for the purpose of simplicity, can be assumed to compute the score and returns a tensor of shape $(N, 1)$, where each element represents the cosine similarity score between each image and text description pair in a batch size of $N$.

  • Then $P(D_i | Q_i)$ is the probability of a certain $ith$ text description being the right one given an image.

This is the tricky part. So the paper says:

In this batch of sentences, only $D_i$ matches the image $Q_i$ ,and treat all other M − 1 sentences as mismatching descriptions.

So in the implementation, one matches one text-description $D_i$ against all other images (for a total of $N$ times). This means to match all $N$ text descriptions against all $N$ images, and we will have a total of $N$ number of cosine similarity scores, each of shape $(N,1)$. This is done by taking one text description, repeating it $N$ number of times, and then matching against all $N$ images to get one batch score of shape $(N,1)$, and then doing this $N$ times. Concatenating the results, we get a tensor of shape $(N,N)$, representing the text-centric losses with the images.

The softmax cross entropy loss is then finally calculated as this:

$$L_1 = - \sum^N_{i=1}\log P(D_i|Q_i)$$

Which technically makes sense. But in the implementation, one would have to compare this resultant $(N,N)$ shaped matrix against some labels tensor for computing the softmax cross entropy loss.

Given such a requirement, would it suffice mathematically to make an identity matrix of shape $(N,N)$ to compute the softmax cross entropy loss?

This is the gap between the research theory and implementation I am facing. Why I think this should be the case: The identity matrix encodes the right class for each column of image captions, given that the rows represent all image features and the columns the text description for some $(N,N)$ tensor - e.g. we have one vector column of shape $(N,1)$ representing just the first text description against all the image features ($N$ rows), i.e. $P(D_0|Q_i)$ for all $i \in N$ - we can use the identity matrix to say that for the $ith$ vector column, only the $ith$ class (indicated by the value 1) is the correct class.

This is a tad different from the usual computer vision classification problem where the label is given beforehand. Because of the image-text pairing, where each image strictly pairs with one text selected, is it right to say that we should use the identity matrix to deal with the cross entropy loss?

This is rather challenging but I would like to see to end this entire implementation to learn something from it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.