PixelCNN++ constructs a model distribution $p(x)$ over images $x\in\mathbb{R}^{n\times n}$ as a product of conditional distributions over pixels

$$p(x)=p(x_1,...,x_{n^2})=\prod_{i=1}^{n^2} p(x_i| x_1,...,x_{i-1})$$

In particular, we choose to model $p(x_i| x_1,...,x_{i-1})$ as a logistic distribution $L(\mu_i, s_i)$ where $\mu_i, s_i = neural\_net_\theta(x_1,...,x_{i-1})$ for parameters $\theta$. The authors claim that $p(x)$ is a probability density distribution, i.e.

$$\int_{\mathbb{R}^{n\times n}} p(x) \; dx= \int_{\mathbb{R}^{n\times n}} \prod_{i=1}^{n^2} L( neural\_net_\theta(x_1,...,x_{i-1})) \; dx=1$$

Q1. How can we guarantee that this integral is 1 for any neural network?

Q2. Is it true for any continuous function?


The answer to both questions follows basic probability definitions. Consider a discrete distribution over $\mathbf{x}=(x_1,x_2)\in\{0,1\}^2$. The equation then becomes

$$ \begin{align} \sum_{ \mathbf{x} \in \{0,1\}^2 } p(\mathbf{x}) &= \sum_{ x_1 \in \{0,1\} }\sum_{x_2\in\{0,1\}} p(x_1)\cdot p(x_2|x_1) \\ &=\sum_{ x_1 \in \{0,1\} }p(x_1)\cdot \sum_{x_2\in\{0,1\}} p(x_2|x_1) \\ &=\sum_{ x_1 \in \{0,1\} }p(x_1) \cdot 1 \\ &=1 \end{align}$$

So the answer to both questions

A1. Expand the equation using basic definitions of probability. Start with simpler example as done above for clarity.

A2. As long as the continues function allows us to use the above proof we are fine.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.