Language Models with Transformers states:

Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling.

A figure from the paper showing pure Transformer architectures (green) performing poorly:

enter image description here

The paper further argues:

Together with positional encoding, Transformers are able to capture long-range dependencies with vague relative token positions. This results in a coarse-grained sequence representation at sentence level. Recent works such as GPT (or GPT- 2) (Radford et al., 2018, 2019) and BERT (Devlin et al., 2018) show that the representations learned on large-scale language modeling datasets are effective for fine-tuning both sentence-level tasks, such as GLUE benchmark (Wang et al., 2018), and token-level tasks that do not rely on word order dependency in the context, such as question answering and NER.

I find this a bit confusing, because the original Transformer paper demonstrated it on natural language translation, where word order is certainly important.

Why are Transformers “suboptimal” for language modeling but not for translation?


There is an experiment you can try yourself. PyTorch includes a language model example, and suggests a "good" LSTM model, which you can train with

python main.py --cuda --emsize 650 --nhid 650 --dropout 0.5 --epochs 40

(This takes about 1 hour)

One can train a Transformer model instead by adding --model Transformer, and also change the initial learning rate, the dropout rate, and the number of attention heads. Still, I tried, but I could not get anything anywhere close to the above LSTM result (validation perplexity around 100) using the transformers (The size of the model seems to be comparable).

The best choice of these parameters that I found was

python main.py --model Transformer --cuda --emsize 650 --nhid 650 --dropout 0.2 --epochs 40 --lr 3

which results in validation perplexity of 163.5.

  • 2
    $\begingroup$ The paper on arXiv is not peer reviewed, so it should be taken with a grain of salt. $\endgroup$ Apr 30, 2021 at 1:06
  • $\begingroup$ @AryaMcCarthy You're right, but see the edit. $\endgroup$
    – MWB
    Apr 30, 2021 at 3:39
  • $\begingroup$ For your experiment, did you tune the hyper parameters? Also, it looks like a toy example, so I wouldn’t take it really seriously. $\endgroup$
    – Tim
    Apr 30, 2021 at 7:28
  • $\begingroup$ @Tim I tried 10-20 different settings. See the update. $\endgroup$
    – MWB
    Apr 30, 2021 at 7:44
  • 1
    $\begingroup$ on the other hand, one of the authors is Alex Smola, who is a very bright chap and has done excellent work in the past, but on the third hand, peer-review is only the first step in establishing the quality of a piece of work, so peer-reviewed papers should be taken with a grain of salt as well ;o) $\endgroup$ Apr 30, 2021 at 7:46

1 Answer 1


I think I acquired some insights into this question after posting it 1.5 months ago, and since there are no other answers, I'll share them:

Plain RNNs are, in practice, incapable of learning long-term dependencies, and while LSTMs can do it, they are still focused on recent inputs. This suits LMs just fine, because LMs are evaluated via PPL and similar scores, under which recent past is extremely informative.

Why are people using Transformers then for LMs, despite this? Two reasons:

  1. Memory consumption and efficiency (Transformers are still efficient with small batch sizes, while large batch sizes use a lot of memory in both LSTMs and Transformers)
  2. Human perception of the quality of generated text is different from PPL. Researchers are actually trying to get their models to pay more attention to less recent past, which is where Transformers are better.

So why did Transformers beat LSTMs on translation then? Two more reasons:

  1. The BLEU score, used to evaluate translations, is different from PPL
  2. Translation needs non-local attention more than plain LMs do (Texts get re-ordered significantly, when translated)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.