Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
$\begingroup$
$\endgroup$
4
-
8$\begingroup$ Someone thought it would be a fun name... $\endgroup$– TimCommented Aug 26, 2021 at 9:37
-
4$\begingroup$ Yeah, +1 to Tim. It’s marketing. $\endgroup$– Arya McCarthyCommented Aug 26, 2021 at 16:18
-
2$\begingroup$ Probably because transformer architectures use series of (non)linear transformations, i.e., recall Q, V, K matrices. That might be the reason. @Tim yes fun name, if we associate with Transformers franchise. Interesting that name is not disputed by Hasbro. $\endgroup$– patagonicusCommented Sep 9, 2021 at 3:19
-
$\begingroup$ But those matrix multiplications are exactly linear. There is only softmax function applied on the top of scalar products of K and Q and then again linearly multiplied by V. $\endgroup$– VladCommented Aug 13, 2023 at 19:17
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Transformer, becuase it uses the attention mechanism with softmax transformation after that using the feedforward with nonlinear transformation. In short it uses different transformations(activation functions) to transform the input from intial representation into final representation if we would explain that in very simple words.
-
17$\begingroup$ ...kinda like every other machine learning or even mathematical model $\endgroup$– AlexCommented Mar 30, 2023 at 10:34
$\begingroup$
$\endgroup$
In mathematics, a transformation is a function that maps from a set into that same set, not into a different set. (In other words, the domain and codomain, or the input space and output space, are the same.)
The mappings performed by most kinds of neural-network layers, e.g. fully connected layers or convolutional layers, are in general not transformations because the sets (feature spaces [definition 1, definition 2]) between which they map are different due to having different numbers of dimensions; and in a stricter sense, maybe even different meanings can be a criterion (e.g. in language processing, certain feature values that represent the word "cat" in one feature space in general don't reresent the word "cat" in another feature space with the same number of dimensions).
On the other hand, performing weighted sums of entire feature vectors (as opposed to weighted sums of individual features) and (nonlinear) rescaling of feature vectors can be considered a transformation due to staying in the same feature space (same number of dimensions and maybe (depending on the sum weights and scaling types) even same/similar meanings of feature values).
The Transformer (machine-learning sense, uppercase "T"!) might be named after such transformations that happen inside them.
Unfortunately, that's a bit misleading because they are biased towards certain kinds of transformations, whereas other (hypothetical) neural-network layers that perform transformations and are biased towards other kinds of transformations cannot use the occupied "Transformers" name anymore. It's a bit like selling apple juice under a brand name "Juice™", and then carrot juice can't be called "juice" anymore without causing confusion.
Maybe other meanings of the following words can explain that name better.
- "transformer" [ Wikipedia, Wiktionary ]
- "transformers" [ Wikipedia ]
- "transformation" [ Wikipedia, Wiktionary ]
- "transform" [ Wikipedia, Wiktionary noun and verb ]
