1
$\begingroup$

I see in this document on variational inference that (p.5):

$L_k = \int q(z_k)E_{-k}[\log(p|z_{-k},x)]dz_k - \int q(z_k)\log q(z_k)dz_k$

It is stated that taking the derivative with respect to $q(z_k)$ results in:

$\frac{dL_k}{dq(z_k)}=E_{-k}[\log(p|z_{-k},x)] - \log q(z_k) - 1$

It looks to me like it has been argued that:

$\frac{d}{dq(z_k)}\int q(z_k)E_{-k}[\log(p|z_{-k},x)]dz_k = \frac{d}{dq(z_k)}[q(z_k)E_{-k}[\log(p|z_{-k},x)]] = E_{-k}[\log(p|z_{-k},x)]$

and

$\frac{d}{dq(z_k)}[- \int q(z_k)\log q(z_k)dz_k]=-\frac{d}{dq(z_k)}[q(z_k)\log q(z_k)]= -\log q(z_k) - 1 $

I can't see why this would work - it looks like there's been a bonus differentiation by $dz_k$. Can someone fill me in on why this is valid?

$\endgroup$

2 Answers 2

2
$\begingroup$

It works because these are functional derivatives.

$\endgroup$
0
$\begingroup$

I just came here having the same question. After spending some time to figure out what the funtional derivatives is, I find that the core idea here is the "local changes of the funcion" in Calculus of Variations and functional derivatives.

In this question, $L_k[q(z_k)]$ is a functional, whose input is the function $q(z_k)$. We want to maximize the functional $L_k[q(z_k)]$ by changing the input function $q(z_k)$. This procedure is called Calculus of Variations.

To maximize the functional $L_k[q(z_k)]$, we can make some small change or variation to the function $q(z_k)$ at a specific point to see how this change or variation can effect the functional.

Let's take another example to make it more clear about what the "local changes" means. Imagine you have a functional that represents the length of a path between two points, A and B, in a plane. The functional could be expressed as:

$ L[path] = \int_A^B \sqrt{1 + \left( \frac{dy}{dx} \right)^2} dx $

  • Local Change: If you nudge the path up or down slightly at some point $x_0$, this is a local change. You are not altering the entire path, just a small segment around $x_0$.

  • Effect on the Functional: The functional derivative $ \frac{\delta L}{\delta y(x)} $ tells you how the total length of the path changes in response to these local adjustments at each point along the path.

So, while calculating the $\frac{dL_k}{dq(z_k)}$, we are interested in the local impact of changing $q(z_k)$ at some specific points.

Therefore, when differentiating with respect to $q(z_k)$, we're looking at the impact of changing $q(z_k)$ at each individual point, and thus the integral sign is omitted, leaving only the integrand evaluated at that point. The result is a function (not an integral) that describes how $ L_k $ changes in response to a small change in $ q(z_k) $ at each point $ z_k $.

Finally, the derivatives of the functional can be derived by only differentiating the integrand, and then regard the $q(z_k)$ as a common variable to do the differentiation.

I figure out that with the help of ChatGPT4. The conversation can be found at the following link:

https://chat.openai.com/share/07c67ad3-f51e-45c0-9f30-73075a632a02

$\endgroup$
3
  • $\begingroup$ Dis you generate the whole answer with ChatGPT? $\endgroup$ Nov 21, 2023 at 8:29
  • $\begingroup$ @User1865345 Of course not. The ChatGPT is only used to help me understand the concept. The answer is written by myself. $\endgroup$
    – yang piao
    Nov 21, 2023 at 15:42
  • $\begingroup$ Please note that ChatGPT and similar Large Language Models are often highly unreliable in formulating coherent statistical assertions. Apropos to that, it is a common precedent here to abstain from using such to create a post, and to tread with caution here. $\endgroup$ Nov 21, 2023 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.