Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling.
A figure from the paper showing pure Transformer architectures (green) performing poorly:
The paper further argues:
Together with positional encoding, Transformers are able to capture long-range dependencies with vague relative token positions. This results in a coarse-grained sequence representation at sentence level. Recent works such as GPT (or GPT- 2) (Radford et al., 2018, 2019) and BERT (Devlin et al., 2018) show that the representations learned on large-scale language modeling datasets are effective for fine-tuning both sentence-level tasks, such as GLUE benchmark (Wang et al., 2018), and token-level tasks that do not rely on word order dependency in the context, such as question answering and NER.
I find this a bit confusing, because the original Transformer paper demonstrated it on natural language translation, where word order is certainly important.
Why are Transformers “suboptimal” for language modeling but not for translation?
There is an experiment you can try yourself. PyTorch includes a language model example, and suggests a "good" LSTM model, which you can train with
python main.py --cuda --emsize 650 --nhid 650 --dropout 0.5 --epochs 40
(This takes about 1 hour)
One can train a Transformer model instead by adding
--model Transformer, and also change the initial learning rate, the dropout rate, and the number of attention heads. Still, I tried, but I could not get anything anywhere close to the above LSTM result (validation perplexity around
100) using the transformers (The size of the model seems to be comparable).
The best choice of these parameters that I found was
python main.py --model Transformer --cuda --emsize 650 --nhid 650 --dropout 0.2 --epochs 40 --lr 3
which results in validation perplexity of 163.5.