I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens.

But it is not clear to me how this kind of model should work in terms of the output format. If the outputs are tokens, they could be represented as embeddings, one-hot or maybe indices/ints extracted from the embeddings?

Source codes I found doing something similar are either old or not that straightforward for learning. Some of my questions are:

Can you describe the shape and meaning of the inputs and outputs for such model?

Is it possible to use an embedded output (or the inverse of an embedding layer to output an integer representing a token)?

If the output is one-hot, doesn’t it get too heavy since the total number of tokens would be around 100k to 1m (number of possible english words)?

Are there some tutorials or examples you’d recommend of a tokenizer being trained using keras/tensorflow (hopefully 2.0)?


There is no need to output embeddings nor one-hot vectors. After all, a tokenizer should work with any words, not only those you have in a vocabulary for the one-hot encoding.

Tokenization is usually formualate as a sequence-labeling problem. The input is a character sequence and you label you label each character by assigning one of following classes:

  • Token boundary with space before (T);

  • Token boundary with a space before (B);

  • No boundary (N).

An example of an input-output pair can be the following:

The cat sleeps on a mat.
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.