Is Elmo equivalent to Fasttext+Bi-directional GRU?

Question

From what I have read, Elmo uses bi-directional LSTM layers to give contextual embeddings for words in a sentence. So if I use a bi-directional LSTM/GRU layer over Fasttext representations of words, will it be the same? If not why? (I know that Fasttext works at the sub-word level while Elmo works at character level)

Also, does it make sense to use a bi-directional LSTM/GRU layer over the representations produced by Elmo?

The task that I am working on is extreme multi-label classification of documents.

Answer 1

So if I use a bi-directional LSTM/GRU layer over Fasttext representations of words, will it be the same?

Sep. 27, 2018 the authors published a follow on work to ELMo, "Dissecting Contextual Word Embeddings: Architecture and Representation". They test whether a bidirectional language model (biLM) can be replicated using Transformers, CNNs, and deeper LSTMs.

They find LSTMs are still the best but a Transformer and CNN representation is much faster and uses much fewer parameters while not being much different result wise. Here are some results from the paper:

They suggest that the general biLM approach seems to work regardless of architecture:

We have shown that deep biLMs learn a rich hierarchy of contextual information, both at the word and span level, and that this is captured in three disparate types of network architectures.

Your specific representation of FastText+Bi-Directional GRU wasn't tried in the paper but this would suggest it's plausible. I would guess it will probably achieve some sort of result, 50%-90% as good as ELMo but much faster.

Also, does it make sense to use a bi-directional LSTM/GRU layer over the representations produced by Elmo?

Yes, you can use any type of architecture over the embedding produced by ELMo. Make sure you explore weighing the layers of ELMo and/or taking separate layers and concatenating them, as per the original ELMo paper.

Answer 2

The new idea in the ELMo paper is add to a deep recurrent net a time-distributed linear layer which combines the word representations learnt at each layer. They show that this makes transfer learning more effective. The deep net can be trained unsupervised on a big corpus. Then it can be reused with a small supervised corpus by training the linear layer and a recurrent output layer, while freezing the weights of the deep recurrent net.