In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length len(s). For training a language model, I've seen people come up with the training sequences in several ways:

  • fix a sequence length, say n=5, and chop up the training text by n to form a training set like (s[0:5], s[1:6]), (s[5:10], s[6:11]), ..., where each tuple (x, y) is an input, output pair, and we have a total of len(s)/n training examples. An advantage is we don't have to set a BPTT truncation parameter, because we always unroll the RNN n times and do BPTT exactly (provided n is small enough). This is done, for example by Karpathy.

  • everything else as the above, except the sliding window takes a stride of 1, instead of n. This gives a training set like (s[0:5], s[1:6]), (s[1:6], s[2:7]), ..., with size len(s)-n+1 (which can be much larger than the previous method). This kind of windowing is used in training traditional n-gram language models, for example here.

  • just use sentences in the text as training examples, rather than sequences of fixed length n. This is done for word level RNN language model in this blog post, where a training example looks like ('SETENCE_START Hello world !','Hello world ! SETENCE_END'). Since the sentences can be very long, a BPTT truncation parameter is needed; we also get less training data with this method, if the sentences are very long.

Does anybody have good insights as which method to use? I consulted the original paper by Mikolov, but couldn't figure out exactly which method they used.

  • $\begingroup$ I don't buy your argument that you get less training data using method 3. It's the same data in all three cases. Method 2 reuses the data multiple times but that's not the same as having more data. $\endgroup$ – Aaron Mar 21 '17 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.