I was recently reading Y. Bengio's paper on NICE (https://arxiv.org/abs/1410.8516). In the paper, authors have taken a view that a good representation involves easy learning of the data distribution. The proposed method therefore transforms the original distribution $p_X(x)$ into $p_H(h)$ using a bijective mapping $f(x)=h$. This is modeled by a neural network containing special type of layer called coupling layer.

Overall, the paper is an interesting read and is very well written. Hence, I decided to implement the model. I coded the equations in section 5 and maximized $log(p_H(h)) + \sum s_i$ using Adam. However, there are certain this that are unclear to me.

  1. How to draw samples from the model? The paper says that ancestral sampling can be used. I'm not sure how to do that. (so sample $h \tilde{} p_H(h)$, then afterwards how can I get $x$?)
  2. Also, how to sample $p_{h_d}$ in the first place , given in section 3.4
  3. How to use this model for inpainting as described in section 5.2.
  4. What is the upper bound of the optimization problem. It seems like its unbounded on upper limit because of $\sum s_{ii}$.
  • $\begingroup$ For part 2: If anyone coding in Tensorflow 2 then sample using tensorflow_probability. Otherwise, the gradients with be None. so don't use scipy.stats $\endgroup$ – Ragnar Nov 2 at 3:38
  1. Since $f$ is bijective, you could also implement its inverse $f^{-1}(h) = x$. For example, to invert the first layer of the network $h_{I_1}^1 = x_{I_1}, h_{I_2}^1 = x_{I_2} + M(x_{I_1})$, you can compute $x_{I_1} = h_{I_1}^1, x_{I_2} = h_{I_2}^1 - m(h_{I_1}^1)$. Section 3.2 describes how to do this for any general coupling layer.

  2. To sample (unconditionally) from the model, you'd simply sample from a standard normal or standard logistic distribution -- most statistical/ml software packages should have a implementation of this. I don't think there's an efficient way to sample conditionally (i.e sampling possible completions of an inpainting task).

  3. Since you can compute $\log p(x)$ (and its gradient) for any data point $x$, you can do gradient descent to find the "most probable image" $\text{argmax}_x \log x$ (although with no guarantees that it's globally optimal). If you split the image up into an observed and unobserved parts $x_o$ and $x_h$, then inpainting is the very similar problem $\text{argmax}_{x_h} \log (x_o;x_h)$. Luckily there's a very simple algorithm called "projected gradient descent" you can use to solve this minimization problem -- simply optimize on $x$ as before, but after every iteration, reset all the observed components of $x$ to their observed values. This is the strategy described in 5.2 (also, they add some random noise after every step, presumably because it helps the optimization escape local minima or something like that).

  1. Consider some small set of datapoints in 1d: $x_1, x_2, \ldots =$ [1,2,2,3,4,7]. You could fit a gaussian $\mathcal{N}(\mu, \sigma^2)$ to this data by maximizing $\log p(x|\mu, \sigma)$, or, if we reparameterized some things, you could define $z = s \cdot x_i$ and use the equivalent log likelihood $\log p(z |\mu, \sigma) + \log s$. Hopefully you can see this will result in the exact same distribution on $x$, and the fact that $s$ was "pulled out" doesn't break anything. This seems like a fairly useless thing to do, but in the NICE paper, by construction, $\sigma^2$ is forced to be 1, so in that case it actually is necessary to get a flexible model.
| cite | improve this answer | |
  • $\begingroup$ Thanks for answering. However, I feel part 1 is still unanswered. Have you checked section 5. Check out the equations. If it was like you mentioned then, it is easy. All the hidden coupling layers are using $x_1$ and $x_2$ for their input to $m$ and not the input to it i.e say $h^{2}_1$ and $h^{2}_2$ as input to $m$ in coupling layer 3 $h^3$. $\endgroup$ – Ragnar Nov 2 at 3:34
  • $\begingroup$ answer 2 is satisfactory. Thanks $\endgroup$ – Ragnar Nov 2 at 3:36
  • $\begingroup$ I'll check solution 3. $\endgroup$ – Ragnar Nov 2 at 3:45
  • $\begingroup$ Also, please give you opinion on 4 in the question. I have designed the network. However, the cost is all negative and unable to get positive value. $\endgroup$ – Ragnar Nov 2 at 3:52
  • $\begingroup$ Also. how to get $b_1$ and $b_2$ in affine coupling layer. A bit confused. $\endgroup$ – Ragnar Nov 2 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.