I am using a Neural Network (ResNet-18) to classify sounds from the UrbanSound8K dataset (https://urbansounddataset.weebly.com/)
As recommended by the dataset creators, I am using 10-fold cross validation using the pre-prepared folds by the creators.
With a neural network, I am also using epochs to train. Each epoch has 10-fold cross validation training (9 folds training, 1 fold validation)
The loss is the categorical cross-entropy.I collect the following stats:
Per epoch average train loss
per epoch average train accuracy
per epoch average valid accuracy
per fold train loss (for example, fold #55 is the 5th fold of the 5th epoch, with 10 folds in each epoch)
per fold train accuracy
per fold validation accuracy
The validation accuracy (per-fold and per-epoch) reaches close to 100% very quickly, within 9 epochs of 10-fold validation in each epoch.
I use all of the data in each fold, for the training and validation processing.
My questions are:
Is there something wrong with my approach?
Is it correct to use epochs with k-fold cross validation using all data in each fold, while training neural networks?
Could it be said that the weights are able to 'remember' the data in between epochs, which is why the network learns so quickly when epochs are used? In consequence, is the approach overfitting the data, given that all data in all folds are being used?
Instead of using full dataset, is it better to use mini-batch samples from the pool of 9-folds to train, reporting validation accuracy on the full dataset of the 10th validation fold, and then reporting average of the validation accuracy in each epoch? (and perhaps, per-fold and per-epoch training loss as well).
However, is the mini-batch approach over many epochs just a slower way than training using the full dataset in all training folds, and eventually lead to the same probably overfitting results?
Please see the tensorboard graphs below for the trends of these stats:
- Validation Accuracy per epoch (total 9 epochs, 99.78%)