In a pet project to see why automatic phonetic transcription is so hard, I would like to use both audio data where each sample is annotated by the sound to which it belongs (a phonetic segments tier) as well as audio data which just comes with un-timestamped phonetic transcription things.
In the tier case, the target data looks like this:
[r r r e e e e p p p p e e a t t e e e e d d d c c l l l a s s s s s e s s s]
(The input sound would have 38 items, as well, the first three of which correspond to the “r” sound, the next four to the “e” sound and so on.) This level of detail is very useful for the model, but hard to obtain, so I want to extend my traning data with cases where I don't have that level of detail. They look like this:
[r e p e a t e d c l a s e s]
and each symbol might correspond to an arbitrary number of subsequent samples of the input audio data.
Obviously, I would like to use both types of data for the learning of my model. Because I want to use audio data as input and the rich, repeating sequence as target for learning, I cannot just use the obvious script to translate tier data into non-repeating data: If I compress tier training data in pre-processing, I lose its richness of information.
This means I would like to have that step be part of the model and be transparent to the optimizer (i.e. probably differentiable).
I assume I could train a seq2seq model on this particular step, but is there a more transparent (and less fiddly) approach to use this kind of compression in a tensorflow stack?
Answers may assume that the classes are represented as ints, using a one-hot encoding, using a learned embedding, or whatever; and the compression step may be implemented implicitly as part of a loss function instead of being an actual translation step.