Run-length encoding on the input sequence of a RNN I have a dataset consisting of sequences of item embeddings a, b, c, ..., in which the length of consecutive runs of an item may be large (comparative to the sequence length). I want to train an RNN that produces a summary of the sequence (by taking the hidden state at the last timestep).
I am currently using a max sequence length L (derived from the data), so that for sequences S of length |S| > L, I am removing the first |S| - L items. Doing this may exclude certain items from the sequences.
I was wondering whether I can apply run-length encoding as a pre-processing step to effectively shorten the sequences and allow for more unique items to be included.
For instance, for this sequence of length 12:
a, a, a, b, b, a, a, c, a, b, b, c

using run-length encoding the following compressed sequence of length 7 is obtained:
(a,3), (b,2), (a,2), (c,1), (a,1), (b,2), (c,1)

Essentially, in addition to feeding the item embedding (a, b, c, ...), an extra feature, the run length, is provided.
Is there an obvious disadvantage / problem using this encoding scheme as a pre-processing stage for training an RNN?
 A: You don't need a neural network to summarize the data! Why not just summarize it? If you need to summarize the "current state" of a sequence that depends on recent items more than on the older ones, you can just use something like exponential smoothing on the one-hot encoded data:
import numpy as np

arr = ["a", "a", "a", "b", "b", "a", "a", "c", "a", "b", "b", "c"]
enc = np.zeros((len(arr), len(set(arr))))

for i, x in enumerate(arr):
    for j, ch in enumerate(set(arr)):
        if x == ch:
            enc[i, j] = 1

def exponential_smoothing(arr, alpha):
    out = np.zeros(arr.shape[1])
    for i in range(arr.shape[0]):
        out = (1-alpha)*out + alpha*arr[i,:]
    return out

exponential_smoothing(enc, 0.5)
## array([0.53125   , 0.08764648, 0.38085938])

If you prefer to smooth the embeddings rather than one-hot codes, you can apply the above procedure to the embeddings. Just apply the embeddings algorithm element-wise to the sequence and then calculate the exponential smoothing on the embeddings.
Implementing it took few minutes, to code, train, and validate a neural network you would need at best several hours. The result of using exponential smoothing is easily interpretable vs using a "black-box" neural network. I bet that even if you used a naive, pure-python implementation it will run faster than the neural network.
