Embeddings/tokenizers for a transformer with binary-valued data

Question

I'm trying to train an encoder-decoder transformer model for completion of binary-valued data. The each input is basically a length-n bitstring $x = (x_1, \dots, x_n) \in \{0,1\}^n$, generated according to some probability distribution, which has been masked to hide a subset of bits during training.

I am confused about what kind of embedding to use for this kind of input/output data. In the case of translation, one would tokenize the input text into $N$ tokens then pre-train an embedding from $\{1, \dots, N\} \rightarrow \mathbb{R}^d$, which seems reasonable when $N \gg d$. But for binary data using a single bit for a token means $N=2$, and so its not clear what the advantage is of embedding such a small alphabet of tokens into some high dimensional space.

What are some justifiable choices for embedding or tokenizing binary-valued data for use in a transformer?

Answer 1

Embedding is a look-up table for efficient representations. For the input embedding, we would like the network to react similarly to synonyms, while in the output embedding, we would like the scores of words that are interchangeable to be similar (Mnih and Teh, 2012). Given that, and given the fact that learning positional embeddings does not improve the performance (Vaswani et al., 2017), we have no use of learned embeddings in the case of binary valued data. Moving on, do we need attention mechanisms so much so that we try to find continuous representations for bits? You could experiment and find out, perhaps first training a simpler model to see the baseline performance.

See A suitable method to predict next binary outcome based on the patterns of past binary data.