Why are Transformers "suboptimal" for language modeling but not for translation?

Question

Language Models with Transformers states:

Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling.

A figure from the paper showing pure Transformer architectures (green) performing poorly:

The paper further argues:

Together with positional encoding, Transformers are able to capture long-range dependencies with vague relative token positions. This results in a coarse-grained sequence representation at sentence level. Recent works such as GPT (or GPT- 2) (Radford et al., 2018, 2019) and BERT (Devlin et al., 2018) show that the representations learned on large-scale language modeling datasets are effective for fine-tuning both sentence-level tasks, such as GLUE benchmark (Wang et al., 2018), and token-level tasks that do not rely on word order dependency in the context, such as question answering and NER.

I find this a bit confusing, because the original Transformer paper demonstrated it on natural language translation, where word order is certainly important.

Why are Transformers “suboptimal” for language modeling but not for translation?

---

Edit:

There is an experiment you can try yourself. PyTorch includes a language model example, and suggests a "good" LSTM model, which you can train with

python main.py --cuda --emsize 650 --nhid 650 --dropout 0.5 --epochs 40

(This takes about 1 hour)

One can train a Transformer model instead by adding --model Transformer, and also change the initial learning rate, the dropout rate, and the number of attention heads. Still, I tried, but I could not get anything anywhere close to the above LSTM result (validation perplexity around 100) using the transformers (The size of the model seems to be comparable).

The best choice of these parameters that I found was

python main.py --model Transformer --cuda --emsize 650 --nhid 650 --dropout 0.2 --epochs 40 --lr 3

which results in validation perplexity of 163.5.

Answer 1

I think I acquired some insights into this question after posting it 1.5 months ago, and since there are no other answers, I'll share them:

Plain RNNs are, in practice, incapable of learning long-term dependencies, and while LSTMs can do it, they are still focused on recent inputs. This suits LMs just fine, because LMs are evaluated via PPL and similar scores, under which the recent past is extremely informative.

Why are some people using Transformers for LMs then, despite this? Two reasons:

1. Memory consumption and efficiency (Transformers are still efficient with small batch sizes, while large batch sizes use a lot of memory in both LSTMs and Transformers)
2. Human perception of the quality of generated text is different from PPL. Researchers are actually trying to get their models to pay more attention to the less recent past, which is where Transformers are better.

So why did Transformers beat LSTMs on the translation tasks then? Two more reasons:

1. The BLEU score, used to evaluate translations, is different from PPL
2. Translation needs non-local attention more than plain LMs do (Texts get re-ordered significantly, when translated)