why is using a mixture of logstic distributions makes sense in pixelcnn++? I went trough the paper and code of the pixelcnn++ model.
From what I understand, they train the network in the following way for predicting the value of a single pixel:
the inputs are the pixel values of the pixels before the pixel we are trying to predict the distribution for. (this happens in the same time for all pixels, from what I understand, but lets stick to one of them).
the network does what is does, and the outputs are the parameters of a mixture of n logistic distribution: pies for the probability of each distribution to be picked, mu for the mean of the distribution and s for the scale of it. 
during training, the network is maximizing the likelihood of getting the correct pixel value under the distribution with the parameters its predicting. 
now, here is what looks weird to me. wouldn't the distribution with the highest probability to produce the correct value is the one where all pies except one are 0 (so only one distribution can be picked), and for that distribution the mean is the pixel value and scale is zero, so only the right value can be picked? and if so, whats the point of predicting a distribution?
in VAE, we use KL loss to get the scale of the normal distribution to be close to 1, so it cant collapse to 0 - but here I don't see anything like this. 
obviously, I don't understand at least one thing about the model - can someone clear this up for me?
 A: You are correct, if the next pixel's intensity could be predicted exactly, then all this fancy mixture density would degenerate into a point estimate (a delta function, essentially). However, that's not always the case.
For example, consider an augmentation that applies small Gaussian noise to every pixel. Now, since the noise is independent for all pixels, knowing previous pixels do not reduce your uncertainty about the current one all the way to zero, hence the optimal distribution does not degenerate, and is more like a Gaussian.
Next, suppose you have a blue car on a while background. This time you use color-adjusting augmentations that sometimes turn the car yellow, red, green or keep the original color. Consider the first colored pixel of a car: at this moment you don't know whether the car is blue, red or whatever. So, at this pixel the optimal distribution is again non-degenerate, and is likely to be multimodal (and thus look like a mixture of unimodal distributions).

But that's about the optimal distributions in the case of infinitely expressive model. In practice, the expressiveness is limited, and the model might be unable to figure out which one of the possible choices it should make.
For example, imagine you're working with human faces, and you're generating their pixels in an autoregressive fashion. Suppose you've generated a couple of pixels of the left eye, and are now starting to generate the right one. For the first one, you have already committed to a certain eye color, but the model might not have the capacity to catch eyes color consistency (in people without heterochromia), especially given their long-range dependency. So even if the color of the right eye is determined by the color of the left one, the model can't catch that (maybe we simply haven't been training it long enough yet), but it can say for sure that the eye can blue, green, brown, but not purple or red. So, in this case it'd preferential to give the model a way to model a multimodal distribution.
