0
$\begingroup$

In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional encoding, given that it is an implementation of such paper? Could I use my own custom positional encodings by adding them to the input sequence, right before using the tf.keras.MultiHeadAttention layer?

Documentation for this layer doesn't say anything about positional encoding.

$\endgroup$
1
$\begingroup$

It doesn't. Positional encoding is not part of the multi-head attention layer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.