Inference in Normalizing Flow model: NICE(non linear independent components estimation)

Question

I was recently reading Y. Bengio's paper on NICE (https://arxiv.org/abs/1410.8516). In the paper, authors have taken a view that a good representation involves easy learning of the data distribution. The proposed method therefore transforms the original distribution $p_X(x)$ into $p_H(h)$ using a bijective mapping $f(x)=h$. This is modeled by a neural network containing special type of layer called coupling layer.

Overall, the paper is an interesting read and is very well written. Hence, I decided to implement the model. I coded the equations in section 5 and maximized $log(p_H(h)) + \sum s_i$ using Adam. However, there are certain this that are unclear to me.

1. How to draw samples from the model? The paper says that ancestral sampling can be used. I'm not sure how to do that. (so sample $h \tilde{} p_H(h)$, then afterwards how can I get $x$?)
2. Also, how to sample $p_{h_d}$ in the first place , given in section 3.4
3. How to use this model for inpainting as described in section 5.2.
4. What is the upper bound of the optimization problem. It seems like its unbounded on upper limit because of $\sum s_{ii}$.

Answer 1

1. Since $f$ is bijective, you could also implement its inverse $f^{-1}(h) = x$. For example, to invert the first layer of the network $h_{I_1}^1 = x_{I_1}, h_{I_2}^1 = x_{I_2} + M(x_{I_1})$, you can compute $x_{I_1} = h_{I_1}^1, x_{I_2} = h_{I_2}^1 - m(h_{I_1}^1)$. Section 3.2 describes how to do this for any general coupling layer.

2. To sample (unconditionally) from the model, you'd simply sample from a standard normal or standard logistic distribution -- most statistical/ml software packages should have a implementation of this. I don't think there's an efficient way to sample conditionally (i.e sampling possible completions of an inpainting task).

3. Since you can compute $\log p(x)$ (and its gradient) for any data point $x$, you can do gradient descent to find the "most probable image" $\text{argmax}_x \log x$ (although with no guarantees that it's globally optimal). If you split the image up into an observed and unobserved parts $x_o$ and $x_h$, then inpainting is the very similar problem $\text{argmax}_{x_h} \log (x_o;x_h)$. Luckily there's a very simple algorithm called "projected gradient descent" you can use to solve this minimization problem -- simply optimize on $x$ as before, but after every iteration, reset all the observed components of $x$ to their observed values. This is the strategy described in 5.2 (also, they add some random noise after every step, presumably because it helps the optimization escape local minima or something like that).

---

4. Consider some small set of datapoints in 1d: $x_1, x_2, \ldots =$ [1,2,2,3,4,7]. You could fit a gaussian $\mathcal{N}(\mu, \sigma^2)$ to this data by maximizing $\log p(x|\mu, \sigma)$, or, if we reparameterized some things, you could define $z = s \cdot x_i$ and use the equivalent log likelihood $\log p(z |\mu, \sigma) + \log s$. Hopefully you can see this will result in the exact same distribution on $x$, and the fact that $s$ was "pulled out" doesn't break anything. This seems like a fairly useless thing to do, but in the NICE paper, by construction, $\sigma^2$ is forced to be 1, so in that case it actually is necessary to get a flexible model.