15
$\begingroup$

If it is for the lack of my effort to search, I apologize in advance but I couldn't find a explicit definition of bits per dimension (bits/dim).

The first mention of its definition I found was from 'Pixel Recurrent Neural Networks'. But it is still quite unclear to me so let me ask.

Defining the 256-softmax output of a image $\boldsymbol{x}$ as $\boldsymbol{y} \in \mathbb{R}^{32 \times 32 \times 256}$, the negative log-likelihood, to my understanding, is $$ - \mathbb{E}_{\boldsymbol{x}} \ln p(\boldsymbol{y}|\boldsymbol{x}). $$ (Note that we are assuming here that image is one-channeled with its size being $32 \times 32 \times 1$.)

According to the above paper (and possibly other materials), it seems to me that the definition of bits/dim is $$ \text{bit/dim} = \dfrac{- \mathbb{E}_{\boldsymbol{x}} \log_2 p(\boldsymbol{y}|\boldsymbol{x})}{32\cdot 32\cdot 1} $$ because it says 'The total discrete log-likelihood is normalized by the dimensionality of the images '.

Questions.

1) Is the above definition correct?

2) Or should I replace $\mathbb{E}_{\boldsymbol{x}}$ by $\sum_{\boldsymbol{x}}$?

$\endgroup$
6
$\begingroup$

It is explained on page 12 here in great detail.

and is also discussed here although in not as much detail.

Compute the negative log likelihood in base e, apply change of base for converting log base e to log base 2, then divide by the number of pixels (e.g. 3072 pixels for a 32x32 rgb image).

To change base for the log, just divide the log base e value by log(2) -- e.g. in python it's like: (nll_val / num_pixels) / numpy.log(2)

and

As noted by DWF, the continuous log-likelihood is not directly comparable to discrete log-likelihood. Values in the PixelRNN paper for NICE's bits/pixel were computed after correctly accounting for the discrete nature of pixel values in the relevant datasets. In the case of the number in the NICE paper, you'd have to subtract log(128) from the log-likelihood of each pixel (this is to account for data scaling).

I.e. -((5371.78 / 3072.) - 4.852) / np.log(2.) = 4.477

$\endgroup$
2
  • $\begingroup$ The second link is not working. $\endgroup$
    – Iordanis
    Feb 10 '20 at 22:50
  • $\begingroup$ I think it was a link to a Reddit discussion on the matter. I can't remember though...maybe here....reddit.com/r/MachineLearning/comments/56m5o2/… $\endgroup$
    – JPJ
    Feb 10 '20 at 22:51
0
$\begingroup$

To add to the answer above, the log-likelihood is your reconstruction loss. In the case of a 256-way softmax it is the categorical cross-entropy.

If you are using tensorflow eg: tf.nn.sparse_softmax_cross_entropy_with_logits the log-likelihood is in natural log so you need to divide by np.log(2.)

If your reconstruction loss is reported as the mean, e.g. tf.reduce_mean you don't need to divide with the image dimensions and/or batch size. On the other hand if it is tf.reduce_sum you will need to divide with the batch size and dimensions of the image.

In case your model is outputting continuous values (e.g. L2 loss) for reconstruction, you are modeling directly a Gaussian Distribution. For that you need to do some transformation, that I am not 100% sure works but is reported at Masked Autoregressive Flow for Density Estimation

$\endgroup$
1
  • $\begingroup$ In PixelRNN (arxiv.org/pdf/1601.06759.pdf), do they divide by log(2)? Did they do it for MNIST too? My results divided by log(2) makes sense for CIFAR-10, but for MNIST I am getting around ~60, and when dividing by log(2) it's around 200 (much larger than 80 as they get). $\endgroup$
    – BBLN
    Sep 19 '20 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.