The following is from Section 2.2 of the Auto-Encoding Variational Bayes paper,

It says the gradient of the lower bound w.r.t $$\phi$$ is a bit problematic because the Monte Carlo estimator exhibits very high variance.

Why? Is it because sampling from $$q_\phi(z|x)$$ is of high variance? What about the gradient w.r.t $$\theta$$? $$\nabla_\theta E_{q_\phi(z|x)}[\log p_\theta(x|z)]=E_{q_\phi(z|x)}[\nabla_\theta\log p_\theta(x|z)]$$ Why is this not described as problematic?

A note to myself
My original confusion was that since the expectation is essentially an integral, $$\int_z q_\phi(z)f_\theta(z)$$ why its derivative w.r.t $$\phi$$ is more difficult than the derivative w.r.t $$\theta$$?

Now I seem to understand that the gradient w.r.t $$\theta$$ can be approximated approximate it by sampling from distribution $$q_\phi(z)$$, whereas $$f_\theta(z)$$ is not a distribution so we cannot do the same for $$\phi$$.

As an alternative we may use the log derivative trick as mentioned in the paper, but it's usually of high variance (as mentioned in the accepted answer).

If I understand correctly, sampling from $q_\phi(z|x)$ is not difficult: it's just a draw from a Gaussian distribution whose mean and variance are specified by $\phi$.
The problem is specifically with marginalizing out $\phi$ using samples from $q_\phi$. Radford Neal has a great blog post on why this can be a bad idea. Apart from the large (possibly infinite) variance of the estimate (assuming finite Monte Carlo samples), the estimate is also very sensitive to tiny changes in $q$'s distribution.
• thank you much, I'm not very familiar with this though. do you mean marginalizing out $z$ using samples from $q_\phi$? and the problem is $q_\phi$ is too narrow (like the posterior in Radford's blog)? so it takes lots of samples to cover the regions where $q_\phi(z|x)$ is small but $q_\phi(z|x)f(z)\nabla\log q_\phi$ can be large? – dontloo Oct 26 '16 at 4:03
• Sorry, yes. Marginalizing out $z$ using samples from $q_\phi$. The gradient of $\log(q_\phi)$ with respect to $\phi$ is $1/q_\phi$, so we're in the same "harmonic mean" territory discussed in the blog post. Values in the tails that are extremely unlikely to show up in a finite sample nevertheless make huge contributions to the final result. – David J. Harris Oct 31 '16 at 18:36