I just started learning about transformers and looked into the following 3 variants

1. The original one from [Attention Is All You Need][1] (Encoder & Decoder)

2. [BERT][2] (Encoder only)

3. [GPT-2][3] (Decoder only)

How does one generally decide whether their transformer model should include encoders only, decoders only, or both encoders and decoders?

As an example, if I want to train a transformer to read a sequence of images of my backend then predict whether it will rain in an hour ("rain" or "not rain"), should this transformer model generally have only decoders?


  [1]: https://arxiv.org/abs/1706.03762
  [2]: https://arxiv.org/abs/1810.04805
  [3]: https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf