RNN(LSTM) model fails to classify new speaker voice

Question

I'm fairly new to ML and at the moment I'm trying to develop a model that can classify spoken digits (0-9) by extracting mfcc features from audio files.

My data set consists of 15 speakers and 2850 training examples (190 audio examples for each digit). I split it into training, and test set and then performed $k$-fold cross validation on the training set.

After fitting the model for 200 epochs the model has achieved around 96% accuracy on both the validation and the test set.

The problem is, when I record my own digit wavs, the model fails to classify them correctly. Why is this happening and what could I do to fix this issue? Is this an example of overfitting?

Extracting mfcc features

  wave, sr = librosa.load(wav_file, mono=True)
  mfcc = librosa.feature.mfcc(wave, sr)
  mfcc = np.pad(mfcc,((0,0),(0,80-len(mfcc[0]))), mode='constant', constant_values=0)

Parameters

learning_rate = 0.0001
batch_size = 64

width = 20  # mfcc features
height = 80  # (max) length of utterance
classes = 10  # digits

Network building

net = tflearn.input_data([None, width, height])
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, classes, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=learning_rate, 
      loss='categorical_crossentropy')

Training

model = tflearn.DNN(net, tensorboard_verbose=0)

kf = sklearn.model_selection.KFold(n_splits=10)

for train,val in kf.split(X):

    X_train, X_val, y_train, y_val = X[train], X[val], y[train], y[val]

    model.fit(X_train, y_train, n_epoch=20, validation_set=(X_val, y_val), 
        show_metric=True, batch_size=batch_size)

print (score_model(X_test, y_test))

Edit

Acc metrics with 1 speaker left out of the data set.

epoch: 050 | loss: 1.18686 - acc: 0.6309 | val_loss: 1.18125 - val_acc: 0.6480 -- iter: 1613/1613 epoch: 100 | loss: 0.76554 - acc: 0.8070 | val_loss: 0.60666 - val_acc: 0.8324 -- iter: 1613/1613 epoch: 150 | loss: 0.43989 - acc: 0.8955 | val_loss: 0.20694 - val_acc: 0.9777 -- iter: 1613/1613 epoch: 200 | loss: 0.38415 - acc: 0.9320 | val_loss: 0.08264 - val_acc: 0.9944 -- iter: 1613/1613

model accuracy (on test data): 0.88

Answer 1

I think there are 2 things going on.

(1) Your recording setup is almost certainly a little different (or a lot different) than the one used to record the original 15 speakers, so the extracted features are different. This could be due to background noise, or the microphone, or interference, or perhaps the accent or speaking style or tenor of your voice is just very different from the 15 speakers.

(2) When testing an additional speaker (either one of the original 15 or yourself) the model does not generalize well. So when adding your voice to the mix, it's different due to (1) and also because the model just doesn't generalize well from 14 speakers to one additional speaker.