This might be a trivial question, but I've come across a paper where some expectation is said to be taken with respect to some pdf. See example:

example of question

How am I to interpret this, and is there some notation to indicate that it should be taken with respect to that specific pdf? I assume that "with respect to" means that it's the function to use within the product inside the summation, but I'm not sure.

For example I interpret ${E}[\log q(z)]$ as ${E}[\log X]$ where $X$ is a random variable distributed according to q, and then make use of the law of the unconcious statitician, where $g(X)$ is $log(X)$, and similarly with ${E}[\log p(z|x)]$, given that x are observed variables. So given my assumption above, the first case should be with respect to $q(z)$, but in the second case it should be with respect to $p(z | X = x)$, so my assumption seems to fail.

Bonus question: Can they break $\log p(x)$ out of the expectation because it is treated as a constant?


This isn’t quite right, $\mathbb{E}[\log p(X)]$ is the expected value of the log of the pdf of the random variable. So if you have a random variable, work out what the density of that value for $p$ is and log it, that’s the expectation you’re considering. Here it explicitly says the expectation is with respect to $q$ but often you will see it in a subscript after the $\mathbb{E}$ when it’s not clear ($\mathbb{E}_q$). And by “respect to” we’re saying that’s how the random variable is distributed and so when we integrate we use $q$.

If you’re wondering about why this is a useful quantity you should have a look at Information Theory, the book by MacKay for instance, and in particular the entropy of a probability distribution.

  • $\begingroup$ So, another super trivial question, what is the difference of saying $q(z)$ and that Z is distributed according to q(z) i.e (Z ~ some pdf that is q(z)), because I'm assuming that the lowercase z is not set? Furthermore, what you're saying is that $E[\log q(z)]=\int \log q(z)*q(z) \mathrm{dz}$ and $E[\log p(z|x)]=\int \log p(z|x)*q(z) \mathrm{dz}$ $\endgroup$ – NotoriousFunk Aug 10 at 8:55
  • $\begingroup$ q(z) is just a pdf, it’s a function, whereas if you say Z is distributed according to q(z) you’re telling me properties about the random variable Z. And yes those two equations are right. $\endgroup$ – XanderJC Aug 10 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.