2
$\begingroup$

In Appendix C.1 of 'Taming VAEs' paper, the authors need to compute the functional derivative $$\frac{\delta}{\delta g\left( z \right)} \mathbb{E}_{q\left(z\mid x\right)} \left[(g \left( z \right) - x)^2 \right].$$ How this can be done explicitly?

$\endgroup$
  • $\begingroup$ @whuber expectation of the square – this is a part of Gaussian log-density $\endgroup$ – Denis Korzhenkov Oct 4 '18 at 7:35
1
$\begingroup$

The functional derivative extends the idea of differentiation to suitable spaces of functions.

The solution is given in the last section below. The intervening sections provide notation, terminology, and concepts for those who might be new to this subject and to make it as clear as possible what assumptions I am adopting to make sense of the question and obtain an answer.

Generalities

Motivation

Recall that the idea behind the derivative is to approximate a function $f:\mathcal{A}\subset \mathbb{R}^m\to\mathbb{R}^n$ in a neighborhood of a point $x\in\mathcal A$ by means of a linear transformation $Df_x:\mathbb{R}^m\to\mathbb{R}^n.$ "Approximate" is understood to mean that as a nonzero displacement $h\in\mathbb{R}^m$ grows small (while $x+h$ remains in $\mathcal A$) the first difference in $f$ is asymptotic to $Df_x$ applied to $h:$

$$f(x+h) - f(x) = (Df_x)h + o(|h|).\tag{1}$$

The notation "$o(|h|)$" refers to some quantity that becomes vanishingly small compared to $|h|$ as $h\to 0.$ The norm of $h$, written $|h|,$ is the Euclidean distance.

Functionals and Functional Derivatives

The analog of $f$ is a functional: it assigns values in $\mathbb{R}^n$ to elements of a space of functions $\Lambda.$ Specifically, given any set $\mathcal S,$ the collection of functions

$$\mathbb{R}^S = \{x:S\to \mathbb{R}\}$$

is a vector space (whose addition operation is pointwise addition of functions and scalar multiplication multiplies the values of functions by constants). When $S$ is finite, $\mathbb{R}^S$ has the same structure as the familiar finite-dimensional vector space.

Whenever $\Lambda \subset S^{\mathbb{R}}$ is a normed sub-vector space and $f:\mathcal{A}\subset\Lambda\to\mathbb{R}^n$ is a function defined on part of $\Lambda,$ formulation $(1)$ still makes sense and defines a derivative of the functional.

Notation and Terminology

The conventions for naming and symbolizing these objects change, though, because there are two tiers of functions in evidence--the elements of $\Lambda$ and the functionals on $\Lambda$--and it could be confusing not to distinguish them explicitly. Here are some of these conventions:

  • Because elements of $\Lambda$ are functions (defined on $S$), they often are still designated with small Latin letters $f,$ $g,$ $h,$ etc.
  • Functions defined on $\Lambda$ are now called "functionals." They may be written as small Greek letters like $\lambda$ or sometimes as capital Latin letters like $F$ or even script letters like $\mathcal L.$
  • The value of a functional $\mathcal L$ at an element $f\in\Lambda$ is written $\mathcal{L}[f]$ with square brackets instead of parentheses.
  • The asymptotically small difference $h$ in $(1)$ is sometimes called a "test function" or "perturbation." Notice that both $f$ and $f+h$ must be in the subspace $\Lambda.$ When this is the case, in order to take limits as $|h|$ grows small, we must require that all $x+\epsilon h \in \Lambda$ for all sufficiently small numbers $\epsilon.$
  • The limiting value of the left hand side of $(1)$ is often informally written "$\delta \mathcal{L}$" as an analog of the differential $\mathrm{d}f$ of the usual Calculus.

Statisticians modify these conventions a little because their functions are random variables, frequently written with capital (sometimes small) Latin letters late in the alphabet: $X$, $Y,$ etc.

Application to the Question

In the context of the question, we may infer the following:

  1. $z$ is a vector-valued random variable. That is, implicitly there is some sample space $\Omega$ and $z$ is a measurable function $z: \Omega\to\mathbb{R}^n.$

  2. $g$ appears to be a (measurable) function $g:\mathbb{R}^n \to \mathbb{R}.$ The notation "$g(z)$" refers to the composite function $$g\circ z:\Omega \to \mathbb{R}.$$ It, too, is a random variable.

  3. The expression $(g(z)-x)^2$ composes the function $h:\mathbb{R}\to\mathbb{R}$ given by $h(y) = (y-x)^2$ with $g\circ z.$ In diagrams we may write $$\Omega \stackrel{z}{\longrightarrow} \mathbb{R}^n \stackrel{g}{\longrightarrow} \mathbb{R} \stackrel{h}{\longrightarrow} \mathbb{R}.$$

  4. The expectation $\mathbb{E}$ is a functional defined on random variables: to each random variable $X:\Omega\to\mathbb{R}$ it assigns a number.

  5. However, since the question concerns taking a functional derivative with respect to $g,$ we need to view this formula as a process $\mathcal L$ that takes any function $g:\mathbb{R}^n\to \mathbb{R}$ and returns an expectation. That is, $\mathcal{L}: \mathbb{R}^{\left(\mathbb{R}^n\right)} \longrightarrow \mathbb{R}$ is given by $$\mathcal{L}[g] = \mathbb{E}[(g(z)-x)^2].$$

The expectation is taken with respect to $z,$ which the notation suggests is given by a distribution named $q(z\mid x).$ Because that distribution will not change, I will drop any explicit mention of it--but you will have to bear this in mind when applying the result to the paper, because this calculation is just part of the calculation it is describing.

Solution

To apply $(1),$ pick an arbitrary test function $h:\mathbb{R}^n\to\mathbb{R}$ and consider the difference

$$\mathcal{L}[g+ h] - \mathcal{L}[g] = \mathbb{E}\left[((g + h)(z) - x)^2\right] - \mathbb{E}\left[(g(z) - x)^2\right].$$

Still motivated by $(1),$ we seek to express this as a linear transformation of $h$ plus obviously small stuff. About the only way to go about that is to apply linearity of expectation and look for algebraic cancellation. Doing this produces

$$\mathbb{E}\left[(2g(z) + h(z) - 2x)h(z)\right] = 2\mathbb{E}\left[(g(z) - x) h(z)\right] + O(|h|^2),$$

(where evidently $|h| = \mathbb{E}(|h|)$ must be the $L^1$ norm).

Since $O(|h|^2) = o(|h|),$ comparison with $(1)$ shows

$$\delta \mathcal{L}_g[h] = 2\mathbb{E}\left[(g(z) - x)h(z)\right].$$

$\endgroup$
  • $\begingroup$ Thanks for your detailed answer! However, in the mentioned paper they obtain the answer $q \left( z \mid x \right) x$, and I still have no idea how (Eq. 1, 9). $\endgroup$ – Denis Korzhenkov Oct 8 '18 at 10:50
  • $\begingroup$ I believe that's because they are doing more than what you described here: the variation occurs within a larger expectation. $\endgroup$ – whuber Oct 8 '18 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.