In a character language model, text is seen as a stream of characters. Say we have a training text as a string
s, with length
len(s). For training a language model, I've seen people come up with the training sequences in several ways:
fix a sequence length, say
n=5, and chop up the training text by
nto form a training set like
(s[0:5], s[1:6]), (s[5:10], s[6:11]), ..., where each tuple
(x, y)is an input, output pair, and we have a total of
len(s)/ntraining examples. An advantage is we don't have to set a BPTT truncation parameter, because we always unroll the RNN
ntimes and do BPTT exactly (provided
nis small enough). This is done, for example by Karpathy.
everything else as the above, except the sliding window takes a stride of
1, instead of
n. This gives a training set like
(s[0:5], s[1:6]), (s[1:6], s[2:7]), ..., with size
len(s)-n+1(which can be much larger than the previous method). This kind of windowing is used in training traditional n-gram language models, for example here.
- just use sentences in the text as training examples, rather than sequences of fixed length
n. This is done for word level RNN language model in this blog post, where a training example looks like
('SETENCE_START Hello world !','Hello world ! SETENCE_END'). Since the sentences can be very long, a BPTT truncation parameter is needed; we also get less training data with this method, if the sentences are very long.
Does anybody have good insights as which method to use? I consulted the original paper by Mikolov, but couldn't figure out exactly which method they used.