I am working on a neural network in which the final output layer will be a Mixture Density Network (MDN), but am confused about the shape of the values that final layer should return.

In the paper in which he introduced MDN's, Christopher Bishop writes: "Note that the total number of network outputs [created by a final MDN layer] is given by (c+2) * m, as compared with the usual c outputs for a network used in a conventional manner" [bottom-most line on page 7].

I have seen some papers that use MDN's, though, that seem to create a different number of outputs. The Master's thesis "Dancing with Deep Learning", for example, has c = 75, m = 25, but the output size is 3775 (2cm + m) [bottom of page 22].

I have also seen Python implementations of MDN layers in Keras in which the output size appears to be (c+2)*m and 2cm + m.

Am I misinterpreting things, or are these implementations genuinely creating different sized outputs. If they are creating different size outputs, what are the motivations for each style output?

Intuitively, I would expect the number of sigmas generated to equal the number of mus, as we need equal shapes to parameterize m Gaussians, but this does not seem to jive with Bishop's (c+2) * m equation.

Edit: I had a talk with two brilliant physicists on campus today, and they helped me see the MDN layer as more of a mathematical construct than a piece of software with a particular architecture. As a mathematical construct, the MDN layer with c sigmas just produces uniform variance around each value in c for a given distribution in the Gaussian mixture. If we find this confining, we can learn a unique sigma for each value in c, which is what the model with output weights 2cm + m above is doing. I believe this solves the riddle.

  • $\begingroup$ What do $c$ and $m$ refer to? $\endgroup$ – user20160 Oct 24 '18 at 13:10
  • $\begingroup$ Sorry, c is the number of outputs the network would predict were there no MDN layer, and m is the number of Gaussians to include in the MDN layer. $\endgroup$ – dduhaime Oct 24 '18 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.