What's the advantage of multi-scale architecture in Normalizing Flow?

Question

While I've read a paper "Density Estimation using Real NVP", I have some confusing parts about multi-scale architecture in Section 3.6.

1. The author said that

We implement a multi-scale architecture using a squeezing operation: for each channel, it divides the image into subsquares of shape 2 × 2 × c, then reshapes them into subsquares of shape 1 × 1 × 4c. The squeezing operation transforms an s × s × c tensor into an s 2 × s 2 × 4c tensor, effectively trading spatial size for number of channels.

I don't exactly understand what's the relationship between "a squeezing operation" and "a multi-scale architecture". I think squeezing operation is just reshaping a size of an input image, but how it could be related with scaling?

2. The author implemented a multi-scale architecture by factoring out inputs whose are directly modeled as Gaussians like this.

The reason that the author implemented like this is that

Propagating a D dimensional vector through all the coupling layers would be cumbersome, in terms of computational and memory cost, and in terms of the number of parameters that would need to be trained.

Although above implementation have an advantage in cost, I think its performance become worse than propagating a D dimensional vector. I am wondering if there is another advantage of above implementation.

I'm very appreciated if you answer the questions. Thank you in advance.

Answer 1

1. Yes, as you exactly said squeezing operation is nothing but reshaping operator. Let me explain multi-scale with dilated convolutions which you can understand it with the GIF below:

   

   Dilated convolution is a specific type of convolution where we expand our support without loss of coverage. Thanks to the expansion via dilation, we can reach further information. This is a way of constructing multi-scale architectures which harness multiple levels information from different contexts.

   In RealNVP, like other type of flows, the problem comes from spatial dimension. If it's too large, it'd be a computational burden. To avoid it, they use divide-by-conquer style approach which is handled by applying a channel-wise mask and squeeze it to reduce the input dimensionality to the coupling layers. After coupling,you can easily invert the reshaping operations. Multi-scale here can also be understood as parallelism in this context. They also explain why they did squeezing operation below:

   At each layer, as the spatial resolution is reduced, the number of hidden layer features in s and t is doubled. All variables which have been factored out at different scales are concatenated to obtain the final transformed output (Equation 16).

2. Well, it may be worse. This depends. The main strength of flow models is also their weakness. Layers need to be designed such that determinant of the Jacobian must be computed efficiently. Also, factoring out the dimensions by half may help learning better. You can think factoring as implicit regularizer. If we would apply the vector with dimensionality D(which is can be high )directly may lead to curse of dimensionality.

Hope this helps.

Answer 2

One of the conceptual advantages of using a multiscale architecture, in my opinion, is ability to model latent factors on different levels of abstraction.

Consider random noise that gets injected near the flow's output. Much of the flow's computation has already been performed, and not much of the layers are left before the output is produced. Thus, we can assume such random noise would only affect some local texture, and would not, for example, introduce another big object to the scene. In contrast, the noise right in the beginning of the flow has all the layers to go through, and it's much easier for a neural network to decode some global properties out of it.

Therefore, multiscale introduces groups of noise variables, and should make it easier to manipulate latent noise vectors. For example, suppose you work with faces and seek noise inputs that define gender and skin color. You are much more likely to find these features near the flow's input than its output.

Also, such grouping is useful for feature extraction, since extracting a full feature vector of the same size as the observation $x$ is not helpful in semi-supervised learning scenarios where only a handful of labeled samples are available.

Answer 3

2. As the author mentioned, the main advantage of the multiscale architecture is the dimensionality reduction. As this is a normalizing flow that requires invertibility, you cannot use traditional operators such as maxpooling, conv2d (bigger than 1×1 filter) etc. to reduce dimensionality. The multiscale architecture helps reduce the dimension while using dimension preserving operators. You can see more on that in CVPR 21's half-day tutorial Normalizing Flows and Invertible Neural Networks in Computer Vision,