I'm reading this paper about mutual information in infoGAN infoGAN_paper_link and already have the code to run it. I pretty much found code for it which is fine and dandy except for the fact that I kinda don't understand some of the code in the cost function. So, I looked at the paper to dissect it for better understanding and came across some math notation that I don't understand (pic below). The usage of the notations I'm trying to figure out are how the "~" in the Expectation subscripts, "'", and "||" symbols are used.

This is what I think the notation means.

  • "~" in the expectation subscript: the variable on the left of "~" can be any continuous value coming from whatever is to the right of the "~"
  • "'" next to the "c" in P(c'|x): I have no clue. I thought those were symbols for derivates but that makes no sense for this equation so it's def not that.
  • "||": I'm not sure. I only know of these symbols being used in Norms but that's obviously not the case here.

Screenshot of a formula from the paper screenshot from the paper

The actual code I was trying to figure out was this in PyTorch. It's the variational lower bound (mutual information term) in the cost (It's not the formulas in the screenshot above). However, I ran into the formulas in the screenshot first before getting to the formula calculations for the variational lower bound formula.

mutual_information_lower_bound = lambda c_true, mean, logvar: Normal(mean,logvar.exp()).log_prob(c_true).mean()

Thanks for the help!


1 Answer 1


It might help to read the introductory chapter(s) of e.g. MIT's Deep Learning Book. They explain the mathematical background of deep learning briefly and, more importantly, introduce the reader to the style of notation commonly found in papers on deep learning.

The three notations you mentioned:

  • $x \sim G(z, \,c)$: $x$ is a random variable distributed as probability distribution $G$ with parameters $z$ and $c$. $c' \sim P(c \, |\, x)$ is also used, which means $c'$ is distributed as $P$ with parameters $c$ conditional on $x$;
  • $c'$ (read: $c$ prime): I think they are recycling notation here and use $c'$ to mean 'alternative' $c$;
  • $D_{KL}(P \, || \, Q)$: The Kullback-Leibler divergence from $Q$ to $P$. This is a measure of 'distance' going from one distribution to another. In the paper, the distance from the predictions made by the model to the true values it is the objective being minimized.

Ian J. Goodfellow, Yoshua Bengio and Aaron Courville (2016). Deep Learning. MIT Press, 2016.

  • $\begingroup$ I'm familiar with some notation when I read papers but I run into more unfamiliar notations as I keep reading more and more so that link will help plenty. Thanks. Is "||" specifically only used for KL-divergence because I can see that it's the expectation of the log of the difference between Q and P. I was wondering if "a||b" i.e. maybe represents log(a/b)? $\endgroup$
    – zipline86
    Apr 12, 2021 at 11:43
  • $\begingroup$ $D_{KL} (A \, || \, B)$ represents the sum (or integral) of $A(x) \log \frac{A(x)}{B(x)}$, this is an important difference from simply $\log \frac{A(x)}{B(x)}$, because the multiplication with $A(x)$ is what makes the KL-divergence asymmetric. $\endgroup$ Apr 12, 2021 at 11:52
  • $\begingroup$ I gotcha on that one. Just to clarify, I meant that I was wondering if something like (A||B) exists without the $D_{kl}$ part? Maybe I'm digging myself into a deeper hole of confusion in what I'm trying to as. I will stop here if my question makes no sense haha $\endgroup$
    – zipline86
    Apr 12, 2021 at 12:01
  • 1
    $\begingroup$ Not as far as I know, there have been some questions about the notation over at math.SE as well: 1, 2. One user suggested it is to emphasize the asymmetry, but gave no reference. The only other use I could find where the double bars are between two arguments is in the study of prime numbers: 3, 4. $\endgroup$ Apr 12, 2021 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.