# Calculating test-time perplexity for seq2seq (RNN) language models

To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$.

My question is: How are these terms computed for a seq2seq language model (say using LSTMs)?

At training time, we encode $s$ as a vector $\mathbf x_s$, initialize the decoder with $\mathbf x_s$ (usually the last hidden state $\mathbf h_n$), provide it with the ground truth sequence $w_1,\dots,w_n$ as input and obtain a prediction $\hat w_1,\dots,\hat w_n$ which goes into a loss function, e.g. cross-entropy.

At test time, I assume:

1. Initializing the decoder with $\mathbf x_s$ would be cheating as it encodes information about upcoming words.
2. We cannot take an encoder hidden state $\mathbf h_i$ and use it as the decoder hidden state $\hat{\mathbf h}_{i}$ at some time $i>1$ since encoder and decoder weight matrices have learned different dynamics.

So my question boils down to: How can we get a good but fair decoder hidden state $\hat{\mathbf{h}}_i$ to predict word $\hat w_i$? How can it encode as much as possible about the history? One option that I see is:

1. Run the encoder to obtain $\mathbf h_1,\dots,\mathbf h_n$.
2. To obtain $\hat{\mathbf h}_{i-1}$, feed $\mathbf h_{i-1}$ as initial hidden state to the decoder, run it until time step $i-1$ always providing the ground truth as input.
3. Compute $\hat{\mathbf h}_i$ using $\hat{\mathbf h}_{i-1}$ and $w_{i-1}$.
4. Obtain $P(w_i|w_1,\dots,w_{i-1})$ by feeding $\hat{\mathbf h}_i$ into the softmax.

Can anybody conform that his is the way to do it?

Alternatively, one can of course feed an uninformative, constant/random initial hidden state to the decoder and just provide the ground truth sequence. However, this feels much weaker than the approach described above.

PS: I know that in machine translation the situation is very different as knowing the encoder state when decoding in another language is not cheating. However, there one would not be allowed to get ground truth input at test time.

I think your wording is somehow unclear. If I am not wrong, you are asking how to estimate the sentence probability provided a trained language model and how to set the state at each step in decoding.

To make it simple you can use the bigram method to do it. Rather than
$$P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$$ you just calculate this, $$P(w_1|<s>), P(w_2|w_1),\dots,P(w_n|w_{n-1}) \text{, where <s> denotes the start symbol}$$ And the while decoding the initial state is the hidden state of the encoder and the subsequent state is one of the output of the previous cell(another is the prediction output).

For the code please see refer to this snippet

state = self.cell.zero_state(1, tf.float32).eval(session=session)
char_probas = []
input = np.zeros((1, 1))
for c, char in enumerate(sentence[:-1]):
input[0, 0] = vocab[char]
feed = {self.input_data: input, self.initial_state: state}
[probs, state] = session.run([self.probs, self.final_state], feed)
char_probas.append(probs[0][vocab[sentence[c+1]]])
probability = np.mean(char_probas)


and the char-rnn-tensorflow.

At test time, for decoding, choose the word with highest Softmax probability as the input to the next time step. The perplexity is calculated as

p(sentence)^(-1/N)

where N is number of words in the sentence.