3
$\begingroup$

I am currently trying to automate some identification process of characteristic noise sounds. For acoustic feature, I calculate MFCC. I have downloaded a free MATLAB toolbox from Dan Ellis'es website. As a classifier, I am trying Neural Network patternnet with Softmax transfer function from Neural Network Toolbox in MATLAB. I have built a training file which consists of ~ 200 sounds samples and ~ 200 additional sounds of "anything else" to give NN a sort of counter balance. I extract 13 MFCC coefficients, 13 delta coefficients calculated from MFCC, and 13 delta2 coefficients calculated from delta1. Also, I feed 5 consecutive frames in sequence at once to NN. So all together, I have like 13*3*5 = 195 inputs. I have also 10 outputs, 5 per frame, and 2 per each class.

When I analyse NN's performance and its gradient progress, I am not quite sure if NN is trained correctly. It looks OK at first as the gradient gets low but it doesn't reach its threshold $10^{-6}$. NN simply stops due to validation. When I test the model on another test file, NN seems to pick also other sounds. It's like its training is not done correctly. I did try different configurations such as 195, 390, 780 neurons in one, two, and three layer but it doesn't seem improve the performance.

Perhaps someone could give an advice on how to improve the classification.

Thanks

Below I have provided some screenshots from NN performances.

NN Gradient. 195 inputs, 195 neurons in a hidden layer, 10 outputs

enter image description here

NN Performance. 195 inputs, 195 neurons in a hidden layer, 10 outputs

enter image description here

NN Gradient. 195 inputs, 3 hidden layers, 780 neurons in each hidden layer, 10 outputs

enter image description here

NN Performance. 195 inputs, 3 hidden layers, 780 neurons in each hidden layer, 10 outputs

enter image description here

Improve this question
$\endgroup$
1
  • $\begingroup$ This question is from 2016 and you marked that as duplicate to post from 2018. Shouldn't it be other way around? Unless you don't like the question. But then it should not be marked as duplicate! $\endgroup$ – Celdor Apr 25 '19 at 13:32
1
$\begingroup$

200 samples is likely too little for such a complicated model. Ideally your dataset should be at least 10x bigger. However, here are some things you can try:

Simplify the model

For example try using just (min,max) or (mean,std) summary across your MFCC frames: 2x13=26 features.

Normalize the sounds

Common for spectrogram data is to normalize each frame by subtracting the median or mean, and divide by RMS energy, standard deviation or max.

Use (log) melspectrogram

It can give better performance over MFCC with expressive classifiers like Neural Networks. Simple linear models often struggle with the co-

Use a less data-hungry classifier

Like Random Forests or kNN.

Data augmentation

Expand your data artificially using data augmentation. Common augmentations for audio data include time stretching, frequency response changes, adding white noise.

Unsupervised feature learning

Use unsupervised learning too learn more powerful features. A simple mechanism is spherical k-means on randomly selected spectrogram patches (3x3) to learn convolution kernels.

Improve this answer
$\endgroup$
-1
$\begingroup$

If it stops early due to validation errors instead of your gradient threshold it is because your network starts to overfit underfits.

The absolute best way to improve the situation is to acquire more data. You say to have 2 x 200 sound samples, which doesn't sound like a lot (although that obviously depends on the type of samples).

Improve this answer
$\endgroup$
9
  • $\begingroup$ In fact I think it's underfit (validation error = test error = training error). It's like the model learning nothing from the training set. $\endgroup$ – SmallChess Sep 16 '16 at 9:56
  • $\begingroup$ @StudentT Should I increase a number of neurons or a number of layers, then? What about the training data. Should I also increase it? I though 200 samples of the same event would be enough. $\endgroup$ – Celdor Sep 16 '16 at 10:03
  • $\begingroup$ @Celdor 200 samples is usually too small for a neutral network. bottiger is not entirely wrong, but your model is not overfitting, but underfitting. $\endgroup$ – SmallChess Sep 16 '16 at 10:05
  • $\begingroup$ @StudentT I am not an expert that's why I came here but I was worried about all three lines in the performance plot are almost equal. Just one more question. When you say that model doesn't learn from training, is it because only underfitting or could there be something else? $\endgroup$ – Celdor Sep 16 '16 at 10:18
  • $\begingroup$ @Celdor If your model is not learning anything, the performance should be very close in training, validation and test sets. Notice that your model stops learning very early, that means you don't have enough data. You don't have enough data to fit your model. This is an example of underfitting. $\endgroup$ – SmallChess Sep 16 '16 at 10:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.