For all transformer models that are either trained for text tasks or specialized for time series or simple equation solving, they tokenize and embed numbers rather than using their raw representation. Why is that?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ All of the non-numeric inputs are embedded, so leaving numbers as numbers would require some kind of transformation into an embedded representation. Otherwise, you can’t concatenate the numeric and non-numeric inputs because they have different shapes. The simplest transformation is an embedding. $\endgroup$– Sycorax ♦Commented Feb 22 at 23:26
Add a comment
|