I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens.
But it is not clear to me how this kind of model should work in terms of the output format. If the outputs are tokens, they could be represented as embeddings, one-hot or maybe indices/ints extracted from the embeddings?
Source codes I found doing something similar are either old or not that straightforward for learning. Some of my questions are:
Can you describe the shape and meaning of the inputs and outputs for such model?
Is it possible to use an embedded output (or the inverse of an embedding layer to output an integer representing a token)?
If the output is one-hot, doesn’t it get too heavy since the total number of tokens would be around 100k to 1m (number of possible english words)?
Are there some tutorials or examples you’d recommend of a tokenizer being trained using keras/tensorflow (hopefully 2.0)?