I'm running a neural network to classify audio files into 4 classes. This uses 3300 1min files split roughly evenly across classes. I split this into 80:10:10 train:validation:test sets. This trains for 50 epochs, I save the model from the epoch with the best validation accuracy, then inference on the test data using this model.
However if I run repeats of the network, the accuracy is often wildly different. It may be 98% on one repeat, then 16% on the next, averaging out at about 57% across several repeats. This occurs for train, val and test data. Its as if it often fails to train, but sometimes has a lucky breakthrough.
Using some predefined features and random forests I can get about 83% accuracy with 0.05 StDev so I know the task can be completed consistently to a good accuracy. When I run the neural net to classify just two classes it also consistently performs well (89%).
The network I am using is adapted from this github repo (with added bells and whistles to get accuracy etc), which itself is adapted from the VGG image classifier. I have bug tested everything I can think of, and if I run it using repeats of the same audio (e.g 3 files from each class repeated 100 times) and put these in the val/test data as well it does a great job (99% acc).
Any ideas on what could be causing these huge differences in accuracy and how to fix it? I'm somewhat new to deep learning so may have overlooked simple ideas.