I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
This is working reasonably well for me, but it would be nice to play around with other options. Are there any good alternatives to this?
"Weighted choice sampling" means that you sample each category with some predefined probability, so you basically sample from a categorical distribution. If each category has fixed probability, there is not much else that you can do about. There are some methods for special cases like Gumbel-max trick when the probabilities are unnormalized, tricks to speed up sampling for large number of categories, etc., but those do not seem to be related to your problem, as you do not mention any technical problems that need solving.
What we usually do when sampling from such language models, is we use softmax with temperature (see e.g. the blog post by Andrej Karpathy, this TensorFlow tutorial, or the Deep Learning with Python book by François Chollet for more details). The idea is pretty simple. When you make predictions from the model, you take the logits predicted by the final layer $z_i$ and pass them through softmax function, to transform them to probabilities i.e. $p_i = \exp(z_i) / \sum_j \exp (z_j)$. The thing that we change is we introduce a hyperparameter, the temperature $T$, so that the softmax function becomes:
$$ p_i = \frac{\exp( z_i\,/\,T )}{\sum_j \exp( z_j\,/\,T )} $$
where $T=1$ leads to standard softmax, decreasing it makes the probabilities more extreme, hence more certain, so the samples are closer to the optimal values that would be predicted by the model, while increasing it leads to more diverse, "random" samples. Quoting Karpathy:
Temperature. We can also play with the temperature of the Softmax during sampling. Decreasing the temperature from 1 to some lower number (e.g. 0.5) makes the RNN more confident, but also more conservative in its samples. Conversely, higher temperatures will give more diversity but at cost of more mistakes (e.g. spelling mistakes, etc).
Then, you use those probabilities same way as you would do with the raw probabilities.
See also the paper by Hinton, Vinyals, and Dean for another example of how temperature is used for other purposes.
There are multiple methods for sampling utterances from a trained language model (LM). What you're doing is certainly a valid approach, and fairly modern. Having, I'll just outline a few more approaches here that people have found empirically useful. These are commonly used in large LM such as those found in GPT-2 or RoBERTa.
As a formalism, we assume a max-probability decoding objective. There isn't usually a single best approach; good approaches are heavily task-dependent as well. However, each of these techniques targets failure modes in neural text generation, which can serve as a good heuristic for your own experimentation.
Greedy decoding. At each time step, select the token with the highest probability. Fast, but trivially leads to non-diverse (and often suboptimal!) responses.
Beam search. At each time step, take the top $k$ generated utterances so far; use those as the starting point for search in the next iteration. Addresses the limitations of greedy decoding without blowing up the search space. Can lead to pathologically repetitive/non-diverse responses.
Pure sampling. This is what you're doing -- take a random choice weighted by the probability density generated at each time step. This actually results (empirically) in generated text with a similar token distribution as human-generated text (+ slightly higher perplexity) -- however, there is no guarantee of syntactic/grammatical coherence.
Softmax with temperature. Not a decoding algorithm, but a common trick. This is an extension to the above approaches that redistribute the probability mass used to sample tokens; @Tim has covered it extensively already on this thread.
Top-$k$ sampling. Builds off of weighted-choice sampling by only retaining $k$ words with the highest probability mass at each timestep and then sampling within that distribution. Lower $k$ leads to more generic output; higher $k$ leads to more diverse output. Pure sampling can be thought of as top-$V$ sampling ($V$ = size of vocabulary); greedy decoding is top-$1$ sampling.
Nucleus sampling. Based on a parameter $0 <= p <= 1$, aggregates the smallest set of words that have summed probability mass $p$. Can be thought of as a variation of top-k sampling with dynamic $k$.
Combinations of these are also valid -- top-k sampling is sometimes used with nucleus sampling, for example. You might also notice that softmax with temperature and nucleus sampling are both methods of redistributing the probability mass over the distribution of tokens; as a toy example, temperature decreases and lower p both have the effect of "sharpening" the distribution by dampening (or removing!) the likelihood of sampling rarer tokens.
Length penalty. You can weight the probability scores of a sentence by a function of the length; without this weighting, max-probability decoding methods will favor shorter sentences (joint log-likelihood monotonically decreases as you add more tokens). This is a common scoring function:
$$\text{length_penalty}(Y) = \frac{(5 + |Y|)^\alpha}{(5 + 1)^\alpha}$$ where $0 < \alpha < 1$, with $\alpha = 0$ reverting to vanilla beam search. "Famously" used in Google Translate.
Repetition penalty. Lower the chance of repetition by discounting the scores of previously-generated tokens. Proposed here; a little finnicky empirically.
Min/max length. This is a quick-and-dirty way to ensure that your model generates text of an appropriate length; I used this personally to tune a summarization model.
