6
$\begingroup$

I'm running a neural network to classify audio files into 4 classes. This uses 3300 1min files split roughly evenly across classes. I split this into 80:10:10 train:validation:test sets. This trains for 50 epochs, I save the model from the epoch with the best validation accuracy, then inference on the test data using this model.

However if I run repeats of the network, the accuracy is often wildly different. It may be 98% on one repeat, then 16% on the next, averaging out at about 57% across several repeats. This occurs for train, val and test data. Its as if it often fails to train, but sometimes has a lucky breakthrough.

Using some predefined features and random forests I can get about 83% accuracy with 0.05 StDev so I know the task can be completed consistently to a good accuracy. When I run the neural net to classify just two classes it also consistently performs well (89%).

The network I am using is adapted from this github repo (with added bells and whistles to get accuracy etc), which itself is adapted from the VGG image classifier. I have bug tested everything I can think of, and if I run it using repeats of the same audio (e.g 3 files from each class repeated 100 times) and put these in the val/test data as well it does a great job (99% acc).

Any ideas on what could be causing these huge differences in accuracy and how to fix it? I'm somewhat new to deep learning so may have overlooked simple ideas.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

It means that the neural network has a high variance. It is prone to overfitting, it sometimes picks the right patterns, sometimes the noise. It can be due to using different training data, random initialization, or both. That is why procedures, as employed by yourself, are useful, if you trained and validated it only once you would never know. What this means for you is that you don't really know how the model would behave at prediction time.

You may check the What should I do when my neural network doesn't generalize well? thread for possible solutions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, I use the same training data so perhaps its the random initialization. However, I should have added (now edited to include) that if I inference over my training data each epoch alongside the val/test, the training data accuracy can also remain terrible from start to finish, and sometimes randomly it starts getting better, making some big leaps in just a few epochs. Is this a symptom of overfitting or something else? $\endgroup$
    – Ben Coral
    Commented Mar 31, 2022 at 14:48
  • 3
    $\begingroup$ @BenCoral it might mean many different things, e.g. using a learning rate that is not sufficient. See the second link for more hints. $\endgroup$
    – Tim
    Commented Mar 31, 2022 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.