Distribution $f$ that minimizes $JSD(f||q) + JSD(f||p)$

Question

What can we say about the distribution $f^*$ that is the solution to the following optimization problem:

$$\min_f JSD(f||p)+JSD(f||q) ,$$ where $p,q$ are given distributions over some set, and $JSD$ is the Jensen-Shannon Divergence.

Intuition is that $f$ would tend to be something that approximates both probabilities, "some sort of average".

I tried opening it up to expectations of $KL$-divergences, and opening them up to one big integral, and deriving it (with the lagrangian constraint $\int_{x\in X} f(x) = 1 $) but it didn't get me far.

We can assume $Supp(p) \cap Supp(q) \neq \emptyset $.

Context: I'm trying to understand what happens when I train the GAN formulation with one Generator and 2 discriminators (each with its own datasets, corresponding to $p$ and $q$). The generator step of the training scheme solves essentially the above optimization problem. In the one-discriminator case it's easy: $\min _fJSD(f||p) \iff f^*=p$.

Answer 1

Just set the partial derivative with respect to $f(x_0)$ (for each possible value of $x_0$) equal to 0.

$$JSD(f||p)=\frac{1}{2}D\left(f||\frac{p+f}{2}\right)+\frac{1}{2}D\left(p||\frac{p+f}{2}\right)$$

$$= \frac{1}{2}\underset{x}{\sum}p(x)\ln\left(\frac{p(x)}{\frac{p(x)+f(x)}{2}}\right)+f(x)\ln\left(\frac{f(x)}{\frac{p(x)+f(x)}{2}}\right)$$

$$\partial_{f(x_0)} JSD(f||p)=\partial_{f(x_0)} \frac{1}{2}\underset{x}{\sum}p(x)\ln\left(\frac{p(x)}{\frac{p(x)+f(x)}{2}}\right)+f(x)\ln\left(\frac{f(x)}{\frac{p(x)+f(x)}{2}}\right)$$ $$=\frac{1}{2}\ln\left(\frac{2f}{f+p}\right)$$

See https://www.wolframalpha.com/input/?i=d%2Fdf+%28p+ln%28p%2F%28%28p%2Bf%29%2F2%29%29+%2B+f+ln%28f%2F%28%28p%2Bf%29%2F2%29%29%29%2F2 for proof of that one.

Therefore

$$\partial_{f(x_0)} JSD(f||p) + JSD(f||q) = \frac{1}{2}\ln\left(\frac{2f}{f+p}\right) + \frac{1}{2}\ln\left(\frac{2f}{f+q}\right) = 0$$

To again save you the headache of solving that by hand...

$$f(x) = \frac{1}{6}\left(\sqrt{p(x)^2+q(x)^2 + 14 p(x)q(x)} + p(x) + q(x)\right)$$

Now there's a problem with this derivation. Nowhere did I enforce $\underset{x}{\sum}f(x)=1$. So let's modify that derivation just a bit by defining $f(x) = \frac{f^\prime(x)}{\underset{x}{\sum}f^\prime(x)}$.

$$\partial_{f^\prime(x)} JSD(f||p) + JSD(f||q) = \frac{\partial f(x)}{\partial f^\prime(x)}\partial_{f(x)} JSD(f||p) + JSD(f||q)=0$$

Note the constant factor of $\frac{\partial f(x)}{\partial f^\prime(x)}$ can just be divided out. Hence we arrive at the same conclusion, this time for $f^\prime(x)$.

$$f^\prime(x) = \frac{1}{6}\left(\sqrt{p(x)^2+q(x)^2 + 14 p(x)q(x)} + p(x) + q(x)\right)$$

We can compute our true $f(x)$ by normalizing the above expression to sum to 1 (following the definition of $f(x) = \frac{f^\prime(x)}{\underset{x}{\sum}f^\prime(x)}$).

I really wish that was something pretty, but it seems there's not anything particularly interpretable about the solution to that minimization problem.