1
$\begingroup$

I have a question about the format of the input of the transformers, depicted in the image below: (taken from the [page](https:// blogs.oracle.com/datascience/multi-head-self-attention-in-nlp)).

enter image description here

When is the input X a matrix?

I think in the question answering problem area X is just a vector and in other tasks like recommendation X is a matrix, but I am not sure...

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

X is a matrix as every token in the input sentence is encoded in a vector.

For example, if the input sentence is "I like trains" each token (assume each word is a token) gets transformed to a vector.

So, for example, "I" --> [0.2, 0.1, 0, 0], "like" --> [0.1, 1, 0, 1], "trains" --> ...

Now you have a 3x4 matrix.

Than, positional encoding is added (without changing the matrix shape) and you get your X input matrix

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.