7
$\begingroup$

Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.

$\endgroup$
4
  • 6
    $\begingroup$ Someone thought it would be a fun name... $\endgroup$
    – Tim
    Commented Aug 26, 2021 at 9:37
  • 2
    $\begingroup$ Yeah, +1 to Tim. It’s marketing. $\endgroup$ Commented Aug 26, 2021 at 16:18
  • 2
    $\begingroup$ Probably because transformer architectures use series of (non)linear transformations, i.e., recall Q, V, K matrices. That might be the reason. @Tim yes fun name, if we associate with Transformers franchise. Interesting that name is not disputed by Hasbro. $\endgroup$ Commented Sep 9, 2021 at 3:19
  • $\begingroup$ But those matrix multiplications are exactly linear. There is only softmax function applied on the top of scalar products of K and Q and then again linearly multiplied by V. $\endgroup$
    – Vlad
    Commented Aug 13, 2023 at 19:17

1 Answer 1

2
$\begingroup$

Transformer, becuase it uses the attention mechanism with softmax transformation after that using the feedforward with nonlinear transformation. In short it uses different transformations(activation functions) to transform the input from intial representation into final representation if we would explain that in very simple words.

$\endgroup$
1
  • 12
    $\begingroup$ ...kinda like every other machine learning or even mathematical model $\endgroup$
    – Alex
    Commented Mar 30, 2023 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.