0
$\begingroup$

I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different tasks.

But arent the layers at the top non linear combinations of the layers at the bottom? So in a model like elmo are we taking linear combinations of non linear outputs from different layers? And dont we do this by default in any normal deep network? Per my understanding elmo allows us to individually cherrypick weights for a given layer for any given task. Is this correct?

$\endgroup$

1 Answer 1

1
$\begingroup$

But arent the layers at the top non linear combinations of the layers at the bottom?

Indeed, this is the case, but that is kind of the point. Although the comparison is not perfect, think of a a large convolutional neural network. The bottom layers learn to distinguish very detailed local features, while the higher level layers learn more abstract information. In this case also, one would expect the different LSTM layers to learn different types of features.

So in a model like elmo are we taking linear combinations of non linear outputs from different layers?

It is important to understand that ELMO is first pre-trained in an unsupervised manner on a large dataset. The task it is trained on is 'language modeling', which essentially means ELMO learns to predict what the next word will be in a particular sentence (i.e. in a specific context). After training ELMO for a long time, it gets really good at this task. After the training is finished, this ELMO model can then be used for other tasks: you feed a particular sentence to ELMO and then indeed take a linear combination of the outputs of the different layers of ELMO (such outputs can be considered context-specific features).

And dont we do this by default in any normal deep network?

In any deep network you indeed learn features for a particular task. The difference is here that you pre-train ELMO on a completely different task and dataset and then reuse that knowledge for a new task by taking a combination of the output of the layers.

Per my understanding elmo allows us to individually cherrypick weights for a given layer for any given task.

That is one way of putting it. Slightly more accurate though would be to say that ELMO provides informative features that are specific to the context of a particular sentence that can be used as an (extra) input to train any given NLP task.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.