3
$\begingroup$

I still don't understand how we can approximate the gradient of an expected value... Indeed it's impossible to sample points and then to average the gradients of them as we have only samples... (How to compute derivatives of samples...?)

The log-derivative trick seems to resolve this issue, and i have read that it allows you to compute Monte Carlo estimate on expressions that were untractable before...

If we recall the formula :

I agree that it's impossible to track the first expression with a Monte Carlo as the gradient of p(theta) is not a distribution. But why is it now possible to track the expectation of p(theta) * grad(log(p(theta))) ? What is the crucial changement ?

Thanks a lot for your potential answers !

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ I also struggle with understanding this but I came up with my own understanding which is as follows: In the first case when we approximate the expectation through averageing samples we would do something like SUM_(f(x)_i ) here every f(x)_i sample does not have any information about the underlying probability which therefore cannot be used to calculate any gradients. With the log trick our approximation becomes SUM_(f(x)_i * log_(p(theta)) ) where now every sample has the information of the probability and therefore we can take the derivative of every sample with respect to theta. $\endgroup$
    – Marcel_marcel1991
    Nov 8, 2019 at 12:31
  • $\begingroup$ Thank you for your answer. It gives me a vague intuition indeed... but it is still not enough to understand it mathematically. $\endgroup$
    – Tbertin
    Nov 13, 2019 at 19:40
  • 1
    $\begingroup$ Did you mean $p(x, \theta)$? Then the gradient could be with respect to $\theta$ while the expectation is taken over $x$. This is the formulation for SGD-based variational Bayes, for example. $\endgroup$
    – deasmhumnha
    Nov 14, 2019 at 2:09
  • 1
    $\begingroup$ I do not know anything about MC but in other contexts like reinforcement learning they do the same trick. I think the crucial point is that one can write the RHS integral over $\theta$ as an expectation (but with respect to another random variable!) because now there is a term $p(\theta)$ which is not there in the left integral. However, if something is an expectation then we can approximate by using the law of large numbers and so forth. $\endgroup$
    – Fabian Werner
    Nov 15, 2019 at 9:20
  • 1
    $\begingroup$ There are some errors in the formula. The distribution should be $p(x \mid \theta)$ or $p(x; \theta)$. Also, integration is over $x$; there should be no $d\theta$ term. $\endgroup$
    – user20160
    Nov 15, 2019 at 17:33

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.