What is mode-averaging in wake-sleep algorithm? I was reading Hinton's paper on Deep Belief Nets, A Fast Learning Algo for DBNs. In the introduction section, the authors write:

Section 5 shows how the weights produced by the fast
  greedy algorithm can be fine-tuned using the “up-down” algorithm. This is a contrastive version of the wake-sleep algorithm Hinton et al. (1995) that does not suffer from the
  “mode-averaging” problems that can cause the wake-sleep algorithm to learn poor recognition weights.

However, the authors do not expand on the term "mode-averaging", neither in this paper nor in the original paper on wake-sleep algorithm. What exactly does it mean and how does it make the wake-sleep algorithm to learn poor recognition weights?
 A: I don't know much about these algorithms, but after a quick look into the papers that's what I understood.
There are two stages called "wake" and "sleep". Algorithm alternates between these stages. In the first stage, algorithm learns intermediate representations of the data (compressed, latent vector) and in the second one, it tries to recover input data from the representation. For the forward and backward pass it uses exactly the same set of weights, the only difference is that in backward pass weights will be transposed. The problem here is that optimal set of weights for the forward and backward pass might be quite different and in this case you might say that there are multiple modes (multimodal distribution). In the forward pass, we force weight to move towards one mode (or maybe set of modes), but on the backward pass, we force weight to move to the other mode (maybe some other set of models). By alternating weight changes into two opposite directions we will end up with the middle point for the weight that might be bad for the backward and forward passes.
That's how it might look visually. 

[Image from the "NIPS 2016 Tutorial: Generative Adversarial Networks" by Ian Goodfellow: https://arxiv.org/pdf/1701.00160.pdf]
A: In the below video, Geoffrey Hinton explains exactly what he means by mode-averaging (around the 10 minute mark).
https://www.youtube.com/watch?v=VKpc_z7b9I0
It has to do with the "explaining away" effect, in which two independent hidden causes become dependent based on the visible observations. The wake-sleep algorithm cannot handle this kind of dependence, because it assumes a factorial distribution over the hidden units in a given layer.
I'll also briefly point out that the wake sleep algorithm is designed for a belief network, in which there are two different sets of weights (which are not just the transpose of one another): The recognition weights, and the generative weights.
