2
$\begingroup$

An encoder decoder network of recurrent neural networks can be used to learn the identity function over some set of sequences. If you do this without attention, the output of the decoder can be thought of as a fixed length representation of these sequences. I've been calling this idea a recurrent autoencoder, but I haven't seen it explored by anyone else, yet. This could be very useful for machine learning tasks on sequences, since you can learn from fixed-length vectors. (Especially if you have lots of unlabeled data, but a small amount of labels)

My question is, "Can a similar thing be done with Transformers?" I see that this question is a bit vague, but this is my best effort at asking it in a way that could be answered.

$\endgroup$
1
  • $\begingroup$ This is actually a very old idea. See RAAM (recursive auto-associative memory) from Pollack, 1990. $\endgroup$ Sep 24, 2021 at 4:59

1 Answer 1

1
+50
$\begingroup$

The simple answer would be: yes, it can and BERT already does something like this.

In an RNN, you take the last hidden because it is the state the network is in after reading the entire sentence. In a transformer network, it is not clear what should the special state be.

BERT does that by prepending a special token (they call it [CLS]). BERT is trained on sentence pairs and the vector corresponding to the [CLS] is used to predict if the two sentences are adjacent in a coherent text. When BERT is fine-tuned for downstream tasks, this vector is used as a single vector representation of the input (no matter if the input is a sentence pair or just one sentence).

You can certainly simulate this also in the sequence-to-sequence setup. The simplest way would be using just a vector for the <s> (beginning of a sentence) token instead of the context vector from encoder-decoder attention. However, I doubt it would get you a better representation than BERT and his Transformer friends do.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.