I'm fairly new to ML and at the moment I'm trying to develop a model that can classify spoken digits (0-9) by extracting mfcc
features from audio files.
My data set consists of 15 speakers and 2850 training examples (190 audio examples for each digit). I split it into training, and test set and then performed $k$-fold cross validation on the training set.
After fitting the model for 200 epochs the model has achieved around 96% accuracy on both the validation and the test set.
The problem is, when I record my own digit wavs, the model fails to classify them correctly. Why is this happening and what could I do to fix this issue? Is this an example of overfitting?
Extracting mfcc features
wave, sr = librosa.load(wav_file, mono=True)
mfcc = librosa.feature.mfcc(wave, sr)
mfcc = np.pad(mfcc,((0,0),(0,80-len(mfcc[0]))), mode='constant', constant_values=0)
Parameters
learning_rate = 0.0001
batch_size = 64
width = 20 # mfcc features
height = 80 # (max) length of utterance
classes = 10 # digits
Network building
net = tflearn.input_data([None, width, height])
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, classes, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=learning_rate,
loss='categorical_crossentropy')
Training
model = tflearn.DNN(net, tensorboard_verbose=0)
kf = sklearn.model_selection.KFold(n_splits=10)
for train,val in kf.split(X):
X_train, X_val, y_train, y_val = X[train], X[val], y[train], y[val]
model.fit(X_train, y_train, n_epoch=20, validation_set=(X_val, y_val),
show_metric=True, batch_size=batch_size)
print (score_model(X_test, y_test))
Edit
Acc metrics with 1 speaker left out of the data set.
epoch: 050 | loss: 1.18686 - acc: 0.6309 | val_loss: 1.18125 - val_acc: 0.6480 -- iter: 1613/1613 epoch: 100 | loss: 0.76554 - acc: 0.8070 | val_loss: 0.60666 - val_acc: 0.8324 -- iter: 1613/1613 epoch: 150 | loss: 0.43989 - acc: 0.8955 | val_loss: 0.20694 - val_acc: 0.9777 -- iter: 1613/1613 epoch: 200 | loss: 0.38415 - acc: 0.9320 | val_loss: 0.08264 - val_acc: 0.9944 -- iter: 1613/1613
model accuracy (on test data): 0.88