I'm currently trying to implement a Mixture Density Network (MDN) based off of the original paper here.
Most of the equations seem pretty straight forward but on page 6 (7 of the PDF) equation 23 has a variable 'c' that isn't explained. The only other place I found a mention to a 'c' is at the end of page 7 (8 of the PDF) where it is used as the number of outputs from the network if you weren't using an MDN, but it seems like it would be strange to use that if your outputs are independent.
I found a python implementation here. It seems like the line:
ps = np.exp(-((y - mu)**2)/(2*sig**2))/(sig*np.sqrt(2*math.pi))
is where Equation 23 is used and 'c' looks to be 1 in this case. Is that because this is a single mapping from x to y so the original network only had one output?
That one line is also a little weird because according to equation 23 shouldn't the
(y - mu)**2 portion actually be the dot product of
(y - mu) with itself? I'm definitely confused by the mu vector, is that the vector of kernel means used in that mixture?
Say you have 3 outputs from your non-MDN network (x, y, z). My understanding is you could set up an MDN in different ways, for example:
A mixture of M tri-variate kernels
M * 3 uni-variate kernels, mixture of M kernels for each output
In both these cases would you have the same number of MDN parameters? (M * 3 means, M * 3 variances, M * 3 mixture coefficients)