In a pet project to see why automatic phonetic transcription is so hard, I would like to use both audio data where each sample is annotated by the sound to which it belongs (a phonetic segments tier) as well as audio data which just comes with un-timestamped phonetic transcription things.

In the tier case, the target data looks like this:

[r r r e e e e p p p p e e a t t e e e e d d d c c l l l a s s s s s e s s s]

(The input sound would have 38 items, as well, the first three of which correspond to the “r” sound, the next four to the “e” sound and so on.) This level of detail is very useful for the model, but hard to obtain, so I want to extend my traning data with cases where I don't have that level of detail. They look like this:

[r e p e a t e d c l a s e s]

and each symbol might correspond to an arbitrary number of subsequent samples of the input audio data.

Obviously, I would like to use both types of data for the learning of my model. Because I want to use audio data as input and the rich, repeating sequence as target for learning, I cannot just use the obvious script to translate tier data into non-repeating data: If I compress tier training data in pre-processing, I lose its richness of information.

This means I would like to have that step be part of the model and be transparent to the optimizer (i.e. probably differentiable).

I assume I could train a seq2seq model on this particular step, but is there a more transparent (and less fiddly) approach to use this kind of compression in a tensorflow stack?

Answers may assume that the classes are represented as ints, using a one-hot encoding, using a learned embedding, or whatever; and the compression step may be implemented implicitly as part of a loss function instead of being an actual translation step.

  • $\begingroup$ My background in ML is not very deep yet, until last week the last time I dealt with ANNs was several years ago. Also, I'm new to XValidated and don't know the culture here yet, feel free to edit or comment this question so that I learn how things work here. $\endgroup$ – Anaphory Apr 2 '18 at 20:52
  • $\begingroup$ Why not just write a little preprocessing script that steps through the vector eliminating duplicates? $\endgroup$ – jbowman Apr 2 '18 at 20:52
  • $\begingroup$ I added a sentence to explain that (I want the step to be transparent to the optimizer), but now I feel the question is badly structured. $\endgroup$ – Anaphory Apr 2 '18 at 20:57
  • $\begingroup$ But if it's preprocessing, it is transparent to the optimizer, it seems to me. The optimizer sees the audio data as one input and the non-repeating sequence as the target... obviously I'm not understanding something. $\endgroup$ – jbowman Apr 2 '18 at 21:04
  • $\begingroup$ Ah, that way round! If I reduce repeating input to non-repeating input, I lose valuable information, because the repeated classes correspond one-class-to-one-sample to the samples in the audio input. Learning from tier data should be much more powerful than learning from non-repeating sequence data. $\endgroup$ – Anaphory Apr 2 '18 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.