Distribution $f$ that minimizes $JSD(f||q) + JSD(f||p)$ What can we say about the distribution $f^*$ that is the solution to the following optimization problem:
$$\min_f JSD(f||p)+JSD(f||q) ,$$
where $p,q$ are given distributions over some set, and $JSD$ is the Jensen-Shannon Divergence.
Intuition is that $f$ would tend to be something that approximates both probabilities, "some sort of average".
I tried opening it up to expectations of $KL$-divergences, and opening them up to one big integral, and deriving it (with the lagrangian constraint $\int_{x\in X} f(x) = 1 $) but it didn't get me far.
We can assume $Supp(p) \cap Supp(q) \neq \emptyset $. 
Context: I'm trying to understand what happens when I train the GAN formulation with one Generator and 2 discriminators (each with its own datasets, corresponding to $p$ and $q$). The generator step of the training scheme solves essentially the above optimization problem. In the one-discriminator case it's easy: $\min _fJSD(f||p) \iff f^*=p$.
 A: Just set the partial derivative with respect to $f(x_0)$ (for each possible value of $x_0$) equal to 0.
$$JSD(f||p)=\frac{1}{2}D\left(f||\frac{p+f}{2}\right)+\frac{1}{2}D\left(p||\frac{p+f}{2}\right)$$
$$= \frac{1}{2}\underset{x}{\sum}p(x)\ln\left(\frac{p(x)}{\frac{p(x)+f(x)}{2}}\right)+f(x)\ln\left(\frac{f(x)}{\frac{p(x)+f(x)}{2}}\right)$$
$$\partial_{f(x_0)} JSD(f||p)=\partial_{f(x_0)} \frac{1}{2}\underset{x}{\sum}p(x)\ln\left(\frac{p(x)}{\frac{p(x)+f(x)}{2}}\right)+f(x)\ln\left(\frac{f(x)}{\frac{p(x)+f(x)}{2}}\right)$$
$$=\frac{1}{2}\ln\left(\frac{2f}{f+p}\right)$$
See https://www.wolframalpha.com/input/?i=d%2Fdf+%28p+ln%28p%2F%28%28p%2Bf%29%2F2%29%29+%2B+f+ln%28f%2F%28%28p%2Bf%29%2F2%29%29%29%2F2 for proof of that one.
Therefore
$$\partial_{f(x_0)} JSD(f||p) + JSD(f||q) = \frac{1}{2}\ln\left(\frac{2f}{f+p}\right) + \frac{1}{2}\ln\left(\frac{2f}{f+q}\right) = 0$$
To again save you the headache of solving that by hand...
$$f(x) = \frac{1}{6}\left(\sqrt{p(x)^2+q(x)^2 + 14 p(x)q(x)} + p(x) + q(x)\right)$$
Now there's a problem with this derivation. Nowhere did I enforce $\underset{x}{\sum}f(x)=1$. So let's modify that derivation just a bit by defining $f(x) = \frac{f^\prime(x)}{\underset{x}{\sum}f^\prime(x)}$.
$$\partial_{f^\prime(x)} JSD(f||p) + JSD(f||q) = \frac{\partial f(x)}{\partial f^\prime(x)}\partial_{f(x)} JSD(f||p) + JSD(f||q)=0$$
Note the constant factor of $\frac{\partial f(x)}{\partial f^\prime(x)}$ can just be divided out. Hence we arrive at the same conclusion, this time for $f^\prime(x)$.
$$f^\prime(x) = \frac{1}{6}\left(\sqrt{p(x)^2+q(x)^2 + 14 p(x)q(x)} + p(x) + q(x)\right)$$
We can compute our true $f(x)$ by normalizing the above expression to sum to 1 (following the definition of $f(x) = \frac{f^\prime(x)}{\underset{x}{\sum}f^\prime(x)}$).
I really wish that was something pretty, but it seems there's not anything particularly interpretable about the solution to that minimization problem.
