Is there a seq2seq model that can encode sentences that include numerical values? I am trying to build a seq2seq model that encodes sentences which include numerical values. For example,
Patient's systolic blood pressure was 128.

Conventional seq2seq models (e.g., RNN encoder/decoder) convert each word into an embedding and then model the sequence of embeddings into a vector (hidden state) that is fed to the decoder. However, this sentence includes a number that needs to be regenerated by the decoder and it does not make a lot of sense to treat '128' as a word and include it in the embedding because 1) you can have any number during test time while your corpora likely includes a subset of possible numbers, and 2) it is very inefficient to model 128 as a string as opposed to a number, and 3) if the decoder generates '127' instead of '128', it is still acceptable and this should lead to a small increase in the loss function, while the conventional seq2seq model would penalize an output of '20', '50' or '10000000' (or even a string like 'hello') the same as '127'.
Any ideas how to model numerical values in a more effective way?
 A: You did not say what are you using the seq2seq model for, but if it is machine translation, you usually do not have to worry about the numbers. Long numbers get segmented into subwords and the model easily learns to copy the tokens.
Alternatively, you really want to make sure that the numbers in the input and the output are the same, you can replace the numbers in the source sentence with special tokens, something like NUM-1, NUM-2, etc. The seq2seq model will learn to copy the special tokens and you can then replace them with the original number.
My answer, however, only holds if you want to copy the numbers and not do any reasoning on top of them.
A: You could take the log of each number, then quantize by rounding so that you end up with a reasonable amount of distinct tokens. Using log ensures that the relative difference between any two numbers which get encoded to the same token is upper bounded.
Another approach is to have tokens 0 through 9 and encode large numbers as many tokens. Large language models have been able to learn basic arithmetic, even when they've not been trained to do so, which is encouraging evidence that this approach is ultimately more flexible and powerful.
