So if I use a bi-directional LSTM/GRU layer over Fasttext
representations of words, will it be the same?
Sep. 27, 2018 the authors published a follow on work to ELMo, "Dissecting Contextual Word Embeddings: Architecture and Representation". They test whether a bidirectional language model (biLM) can be replicated using Transformers, CNNs, and deeper LSTMs.
They find LSTMs are still the best but a Transformer and CNN representation is much faster and uses much fewer parameters while not being much different result wise. Here are some results from the paper:
They suggest that the general biLM approach seems to work regardless of architecture:
We have shown that deep biLMs learn a rich hierarchy of contextual information, both at the word
and span level, and that this is captured in three
disparate types of network architectures.
Your specific representation of FastText+Bi-Directional GRU wasn't tried in the paper but this would suggest it's plausible. I would guess it will probably achieve some sort of result, 50%-90% as good as ELMo but much faster.
Also, does it make sense to use a bi-directional LSTM/GRU layer over the
representations produced by Elmo?
Yes, you can use any type of architecture over the embedding produced by ELMo. Make sure you explore weighing the layers of ELMo and/or taking separate layers and concatenating them, as per the original ELMo paper.