5
$\begingroup$

In one of Alex Graves' papers (and several other authors as well) utilize the term bits per character (BPC). The paper that I am referencing here is "Generating Sequences with Recurrent Neural Networks" (http://arxiv.org/abs/1308.0850).

In the paper he defines BPC as $-\log_2 P(x_{t+1} | y_t)$, which is defined in Section 3.1 and results are shown in Table 1.

How is this exactly calculated, especially when it comes to a recurrent neural network, such as a char-rnn? For example, given the input and prediction of a recurrent neural network, how do I compute this bits per character?

Also, this question addresses this (https://stackoverflow.com/questions/17797922/how-to-calculate-bits-per-character-of-a-string-bpc) but none of the answers explain how it is calculated, especially when it comes to a RNN.

$\endgroup$
9
$\begingroup$

From my understanding, the BPC is just the average cross-entropy (used with log base 2).

In the case of Alex Graves' papers, the aim of the model is to approximate the probability distribution of the next character given past characters. At each time step $t$, let's call this (approximate) distribution $\hat{P}_t$ and let $P_t$ be the true distribution. These discrete probability distributions can be represented by a vector of size $n$, where n is the number of possible characters in your alphabet.

So the BPC or average cross-entropy can be calculated as follows:

\begin{align} bpc(string) = \frac{1}{T}\sum_{t=1}^T H(P_t, \hat{P}_t) &= -\frac{1}{T}\sum_{t=1}^T \sum_{c=1}^n P_t(c) \log_2 \hat{P}_t(c), \\ & = -\frac{1}{T}\sum_{t=1}^T \log_2 \hat{P}_t(x_t). \end{align} Where $T$ is the length of your input string.

The equality in the second line comes from the fact that the true distribution $P_t$ is zero everywhere except at the index corresponding to the true character $x_t$ in the input string at location $t$.

Two things to note:

  • When you use an RNN, $\hat{P}_t$ can be obtained by applying a softmax to the RNN's output at time step $t$ (The number of output units in your RNN should be equal to $n$ - the number of characters in your alphabet).
  • In the equation above, the average cross-entropy is calculated over one input string of size T. In practice, you may have more than one string in your batch. Therefore, you should average over all of them (i.e. $bpc = mean_{strings} bpc(string)$).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.