# RNN(LSTM) model fails to classify new speaker voice

I'm fairly new to ML and at the moment I'm trying to develop a model that can classify spoken digits (0-9) by extracting mfcc features from audio files.

My data set consists of 15 speakers and 2850 training examples (190 audio examples for each digit). I split it into training, and test set and then performed $k$-fold cross validation on the training set.

After fitting the model for 200 epochs the model has achieved around 96% accuracy on both the validation and the test set.

The problem is, when I record my own digit wavs, the model fails to classify them correctly. Why is this happening and what could I do to fix this issue? Is this an example of overfitting?

Extracting mfcc features

  wave, sr = librosa.load(wav_file, mono=True)
mfcc = librosa.feature.mfcc(wave, sr)


Parameters

learning_rate = 0.0001
batch_size = 64

width = 20  # mfcc features
height = 80  # (max) length of utterance
classes = 10  # digits


Network building

net = tflearn.input_data([None, width, height])
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, classes, activation='softmax')
loss='categorical_crossentropy')


Training

model = tflearn.DNN(net, tensorboard_verbose=0)

kf = sklearn.model_selection.KFold(n_splits=10)

for train,val in kf.split(X):

X_train, X_val, y_train, y_val = X[train], X[val], y[train], y[val]

model.fit(X_train, y_train, n_epoch=20, validation_set=(X_val, y_val),
show_metric=True, batch_size=batch_size)

print (score_model(X_test, y_test))


Edit

Acc metrics with 1 speaker left out of the data set.

epoch: 050 | loss: 1.18686 - acc: 0.6309 | val_loss: 1.18125 - val_acc: 0.6480 -- iter: 1613/1613 epoch: 100 | loss: 0.76554 - acc: 0.8070 | val_loss: 0.60666 - val_acc: 0.8324 -- iter: 1613/1613 epoch: 150 | loss: 0.43989 - acc: 0.8955 | val_loss: 0.20694 - val_acc: 0.9777 -- iter: 1613/1613 epoch: 200 | loss: 0.38415 - acc: 0.9320 | val_loss: 0.08264 - val_acc: 0.9944 -- iter: 1613/1613

model accuracy (on test data): 0.88

• What happens if you leave out one of the 15 speakers and train on the remaining 14 and test on the left-out speaker?
– Sycorax
Jun 15, 2018 at 15:38
• @Sycorax Although the model predicts with higher confidence for the left out speaker (than my voice), it's not classifying correct most of the times. Jun 15, 2018 at 16:28