I've been having a lot of fun learning keras, and have working code that inputs raw voice amplitudes (usually about 100k of 200 audio samples each) and uses ~ 500 LSTM Nodes with linear regression to predict the next amplitude. For my toy corpus, myself recorded saying 'Hello' 50 times, it is learning and validating well, and can predict valid wavforms sounding pretty good. The problem is, the model has no way to know where it is in the word, and is jumping around from phoneme to phoneme. I have another model, using mfcc, which predicts phonemes with about 85% accuracy. What I'd like to do is pass in this additional phoneme information to the first model (along with the amplitudes), to give the model more context. When I convert the phoneme one hot vector to normalized floats and just append it into the amplitude samples my loss is high and the model is under fitting. I hope this isn't too broad, but essentially how can I feed in different types of data/features?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.