I am working on a neural network in which the final output layer will be a Mixture Density Network (MDN), but am confused about the shape of the values that final layer should return.
In the paper in which he introduced MDN's, Christopher Bishop writes: "Note that the total number of network outputs [created by a final MDN layer] is given by (c+2) * m, as compared with the usual c outputs for a network used in a conventional manner" [bottom-most line on page 7].
I have seen some papers that use MDN's, though, that seem to create a different number of outputs. The Master's thesis "Dancing with Deep Learning", for example, has c = 75, m = 25, but the output size is 3775 (2cm + m) [bottom of page 22].
Am I misinterpreting things, or are these implementations genuinely creating different sized outputs. If they are creating different size outputs, what are the motivations for each style output?
Intuitively, I would expect the number of sigmas generated to equal the number of mus, as we need equal shapes to parameterize m Gaussians, but this does not seem to jive with Bishop's (c+2) * m equation.
Edit: I had a talk with two brilliant physicists on campus today, and they helped me see the MDN layer as more of a mathematical construct than a piece of software with a particular architecture. As a mathematical construct, the MDN layer with
c sigmas just produces uniform variance around each value in
c for a given distribution in the Gaussian mixture. If we find this confining, we can learn a unique sigma for each value in
c, which is what the model with output weights 2cm + m above is doing. I believe this solves the riddle.