I'm fairly new to ML and at the moment I'm trying to develop a model that can classify spoken digits (0-9) by extracting mfcc features from audio files.

My data set consists of 15 speakers and 2850 training examples (190 audio examples for each digit). I split it into training, and test set and then performed $k$-fold cross validation on the training set.

After fitting the model for 200 epochs the model has achieved around 96% accuracy on both the validation and the test set.

The problem is, when I record my own digit wavs, the model fails to classify them correctly. Why is this happening and what could I do to fix this issue? Is this an example of overfitting?

Extracting mfcc features

  wave, sr = librosa.load(wav_file, mono=True)
  mfcc = librosa.feature.mfcc(wave, sr)
  mfcc = np.pad(mfcc,((0,0),(0,80-len(mfcc[0]))), mode='constant', constant_values=0)


learning_rate = 0.0001
batch_size = 64

width = 20  # mfcc features
height = 80  # (max) length of utterance
classes = 10  # digits

Network building

net = tflearn.input_data([None, width, height])
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, classes, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=learning_rate, 


model = tflearn.DNN(net, tensorboard_verbose=0)

kf = sklearn.model_selection.KFold(n_splits=10)

for train,val in kf.split(X):

    X_train, X_val, y_train, y_val = X[train], X[val], y[train], y[val]

    model.fit(X_train, y_train, n_epoch=20, validation_set=(X_val, y_val), 
        show_metric=True, batch_size=batch_size)

print (score_model(X_test, y_test))


Acc metrics with 1 speaker left out of the data set.

epoch: 050 | loss: 1.18686 - acc: 0.6309 | val_loss: 1.18125 - val_acc: 0.6480 -- iter: 1613/1613 epoch: 100 | loss: 0.76554 - acc: 0.8070 | val_loss: 0.60666 - val_acc: 0.8324 -- iter: 1613/1613 epoch: 150 | loss: 0.43989 - acc: 0.8955 | val_loss: 0.20694 - val_acc: 0.9777 -- iter: 1613/1613 epoch: 200 | loss: 0.38415 - acc: 0.9320 | val_loss: 0.08264 - val_acc: 0.9944 -- iter: 1613/1613

model accuracy (on test data): 0.88

  • $\begingroup$ What happens if you leave out one of the 15 speakers and train on the remaining 14 and test on the left-out speaker? $\endgroup$
    – Sycorax
    Commented Jun 15, 2018 at 15:38
  • $\begingroup$ @Sycorax Although the model predicts with higher confidence for the left out speaker (than my voice), it's not classifying correct most of the times. $\endgroup$
    – Moras
    Commented Jun 15, 2018 at 16:28

1 Answer 1


I think there are 2 things going on.

(1) Your recording setup is almost certainly a little different (or a lot different) than the one used to record the original 15 speakers, so the extracted features are different. This could be due to background noise, or the microphone, or interference, or perhaps the accent or speaking style or tenor of your voice is just very different from the 15 speakers.

(2) When testing an additional speaker (either one of the original 15 or yourself) the model does not generalize well. So when adding your voice to the mix, it's different due to (1) and also because the model just doesn't generalize well from 14 speakers to one additional speaker.

  • $\begingroup$ I agree that (1) probably has it's part on this problem but what about the (2)? Why does the model not generalize..? Small training set or... I'm doing something wrong? $\endgroup$
    – Moras
    Commented Jun 15, 2018 at 18:03
  • $\begingroup$ It's probably more to do with the training set than with the model. But it's impossible to say anything definitive -- NNs are incredibly sensitive to the choices you make in architecture, initialization, optimization algorithm, regularization... you might tweak this and get something that generalizes better, but that will require lots of experimentation. $\endgroup$
    – Sycorax
    Commented Jun 15, 2018 at 18:06
  • $\begingroup$ It seems like i can't do much then (given my experience too).. Thank you very much though for your clarification, cleared some things..!! $\endgroup$
    – Moras
    Commented Jun 15, 2018 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.