Bayesian neural networks: very multimodal posterior?

Question:

How do Bayesian treatments of neural networks address the fact that the posterior has an exponentially large number of modes?

Background:

There seems to be a lot of interest in Bayesian treatments of neural networks, where we attempt to model the posterior distribution over network weights given the data, using e.g. Laplace approximation, Monte Carlo, or variational methods. In principle, this would allow you to integrate over model parameters to avoid overfitting and to provide well-calibrated uncertainty estimates for predictions.

For multilayer perceptrons, the posterior has an exponentially large number of symmetrical modes since the parameters are not identifiable. (As pointed out in Kevin Murphy's book "Machine Learning: A Probabilistic Perspective", Chapter 16.5.5, we can permute the identities of any of the hidden units without affecting the likelihood, leading to $H!$ equivalent settings of the parameters, where $H$ is the number of hidden units. If the neural net uses an activation function like $\tanh$ which is an odd function ($-\tanh(x)=\tanh(-x)$), there are also $2^H$ sign-flip degeneracies since we can pick a hidden unit and flip the sign of all its incoming edges as long as we also flip the sign of all its outgoing edges.)

So for even a tiny feedforward net with $H=15$, the posterior will have $>10^{12}$ posterior modes. This sounds like it could be a big problem for Monte Carlo approximations, for example, since there's no way you could draw even one sample from each of the modes. On the other hand, I guess it could be the case that since the posterior modes introduced by parameter unidentifiability all equivalent, you're fine as long as you model at least one of them well...

Is this actually a problem? If so, how can it be addressed?

• The sign-flipping problem can be addressed by pinning one of the indeterminate parameter sets to be positive. The symmetric multimodality problem is only a concern if you care about which hidden node corresponds to what latent features -- otherwise, finding a single mode is sufficient (due to symmetry). – Sycorax Jul 17 '15 at 1:12
• Cool question. I wonder if it might be solved by some kind of ordering of the nodes? The permutations are effectively irrelevant/equivalent, right? So for a single-hidden-layer model, you could order the nodes in the first layer by their incoming weights. Of course, that's only going to work in trivial cases (e.g. one input variable), but maybe there's a way to expand it for more complex structures. – naught101 Feb 25 '16 at 5:19