why does data external from training and testing my neural network perform much worse than statistical accuracy?

Question

I've got a problem with my neural network (used to recognize audio signals, an expansion of the UrbanSound dataset problem): when I fit the model the accuracy of both train and validation is near 90%. Then, using some testing data the accuracy drops to 80%. The main problem is that when I try to classify a file recorded by myself, or some audio from video on youtube: the accuracy drops really low, making really bad predictions.

I don't understand where the problem could be:

I already split the audio in chunks of 1 second (with 80% of overlapping) for training to get similar but different audio samples.

I don't use regularization (L1-L2 or dropout) because it worsen the accuracy results

my neural network has 132 features as input, 2 layers with 100 neurons each and a final layer with 14 labels, and I'm using Keras to build it

I'm fairly new to machine learning and I fear I'm missing something obvious, so I'm up for any advice.

Answer 1

Your problem is probably that the data you are trying to use for prediction (youtube audio and data recorded by you) comes from a quite different distribution then the data on which the algorithm was trained and evaluated. It can be caused by using different microphone, people talking different dialect or audio can contain sounds that didn't appear in trainset nor devset.

The data in your dev/test set should always be as close as possible to the data you care about (in your case probably sounds recorded by you). Otherwise your dev/test set doesn't tell you accurately how your algorithm will perform in your task.

If possible the same should be for your trainset, but you usually don't have so much data and it helps to add to trainset every data you can get from other datasets.

You can read more about this problem in chapter 6 of Machine Learning Yearning book by Andrew Ng.