Can someone explain CAM loss used in U-GAT-IT paper?

Question

I have been reading a recent paper accepted at ICLR, U-GAT-IT, which seems to produce pleasing results in the image-to-image translation tasks. There are four kinds of loss used in this paper: Adversarial loss, Identity loss, Cycle loss, and CAM loss. I could understand all losses except CAM loss. In the paper, CAM loss is defined as such:

CAM loss: By exploiting the information from the auxiliary classifiers $\eta_s$ and $\eta_{D_t}$, given an image $x \in \{X_s, X_t\}$. $G_{s \to t}$ and $D_t$ get to know where they need to improve or what makes the most difference between two domains in the current state:

\begin{align} L_{cam}^{s \to t}&=-(\mathbb{E}_{x \sim X_s}[\log(\eta_s(x))]+\mathbb{E}_{x \sim X_t}[\log(1-\eta_s(x))]), \tag{5} \\\\ L_{cam}^{D_t}&=\mathbb{E}_{x \sim X_t}[(\eta_{D_t}(x))^2]+\mathbb{E}_{x \sim X_s}[(1-\eta_{D_t}(G_{s \to t}(x)))^2]. \tag{6} \end{align}

I couldn't understand the loss used in $(5)$ as shown in the picture above. My naive guess is that minimizing this loss encourages the auxiliary classifier $\eta_s$ to output higher value for the $x$ sampled from source domain $X_s$ and from target domain $X_t$. But, I couldn't reason about the purpose of this.

I get even more confused when I inspect the official implementation of this loss.

fake_A2B, fake_A2B_cam_logit, _ = self.genA2B(real_A)
fake_B2A, fake_B2A_cam_logit, _ = self.genB2A(real_B)

fake_A2A, fake_A2A_cam_logit, _ = self.genB2A(real_A)
fake_B2B, fake_B2B_cam_logit, _ = self.genA2B(real_B)

G_cam_loss_A = self.BCE_loss(fake_B2A_cam_logit, torch.ones_like(fake_B2A_cam_logit).to(self.device)) + self.BCE_loss(fake_A2A_cam_logit, torch.zeros_like(fake_A2A_cam_logit).to(self.device))
G_cam_loss_B = self.BCE_loss(fake_A2B_cam_logit, torch.ones_like(fake_A2B_cam_logit).to(self.device)) + self.BCE_loss(fake_B2B_cam_logit, torch.zeros_like(fake_B2B_cam_logit).to(self.device))

Can someone help me understand CAM loss?

Answer 1

my understanding of this is far from perfect but here is the pieces of information that helped me:

Why do we need this auxiliary classifier?

First in order to understand the role of the auxiliary classifier I read this paper which explain how CAM works: The classifier consist of convolutions, that extract the feature map (red, blue an green layers) those 2D maps are reduced to single values, and a dense layer perform the classification. I won't enter in the details, but this specific architecture allow to locate which portions of the image lead to this classification.

Intuitively we will use this in order to find what portion of image are relevant to tell that the image belongs to class A.

Understanding the code:

Back to U-GAT-IT: the auxiliary classifier is used to tell if an image comes from A or not (resp.B). So if we want to convert from A to B (resp.B to A), we want the auxiliary classifier to return values close to 1 when the image comes from A (resp.B) and close to 0 when images come from B (resp.A). This is easy to do: take a bunch of image from A, run the prediction of auxiliary classifier (cam_logit) fake_A2B, fake_A2B_cam_logit, _ = self.genA2B(real_A) and make them closer to 1: self.BCE_loss(fake_B2A_cam_logit, torch.ones_like(fake_B2A_cam_logit).to(self.device))

Now take images from B, run the same classifier fake_A2A, fake_A2A_cam_logit, _ = self.genB2A(real_A) and make the predictions closer to 0 self.BCE_loss(fake_A2A_cam_logit, torch.zeros_like(fake_A2A_cam_logit).to(self.device))

By doing so you know what portion of each image determines the fact that it comes from A (resp.B). So you know what to change to turn it into an image from B (resp.B).

Link with the maths from the paper:

I found this interesting article that explains the math behind Binary Cross Entropy, if you take the formula, you can unfold it and find the formula from the paper (haven't tried)

I hope this helps !