Understanding math notation in infoGAN paper

Question

I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I kinda don't understand some of the code in the cost function. So, I looked at the paper to dissect it for better understanding and came across some math notation that I don't understand (pic below). The usage of the notations I'm trying to figure out are how the "~" in the Expectation subscripts, "'", and "||" symbols are used.

This is what I think the notation means.

• "~" in the expectation subscript: the variable on the left of "~" can be any continuous value coming from whatever is to the right of the "~"
• "'" next to the "c" in P(c'|x): I have no clue. I thought those were symbols for derivates but that makes no sense for this equation so it's def not that.
• "||": I'm not sure. I only know of these symbols being used in Norms but that's obviously not the case here.

---

Screenshot of a formula from the paper

The actual code I was trying to figure out was this in PyTorch. It's the variational lower bound (mutual information term) in the cost (It's not the formulas in the screenshot above). However, I ran into the formulas in the screenshot first before getting to the formula calculations for the variational lower bound formula.

mutual_information_lower_bound = lambda c_true, mean, logvar: Normal(mean,logvar.exp()).log_prob(c_true).mean()

Thanks for the help!

Answer 1

It might help to read the introductory chapter(s) of e.g. MIT's Deep Learning Book. They explain the mathematical background of deep learning briefly and, more importantly, introduce the reader to the style of notation commonly found in papers on deep learning.

The three notations you mentioned:

• $x \sim G(z, \,c)$: $x$ is a random variable distributed as probability distribution $G$ with parameters $z$ and $c$. $c' \sim P(c \, |\, x)$ is also used, which means $c'$ is distributed as $P$ with parameters $c$ conditional on $x$;
• $c'$ (read: $c$ prime): I think they are recycling notation here and use $c'$ to mean 'alternative' $c$;
• $D_{KL}(P \, || \, Q)$: The Kullback-Leibler divergence from $Q$ to $P$. This is a measure of 'distance' going from one distribution to another. In the paper, the distance from the predictions made by the model to the true values it is the objective being minimized.

---

Ian J. Goodfellow, Yoshua Bengio and Aaron Courville (2016). Deep Learning. MIT Press, 2016.