0
$\begingroup$

I am using a Neural Network (ResNet-18) to classify sounds from the UrbanSound8K dataset (https://urbansounddataset.weebly.com/)

As recommended by the dataset creators, I am using 10-fold cross validation using the pre-prepared folds by the creators.

With a neural network, I am also using epochs to train. Each epoch has 10-fold cross validation training (9 folds training, 1 fold validation)

The loss is the categorical cross-entropy.I collect the following stats:

  • Per epoch average train loss

  • per epoch average train accuracy

  • per epoch average valid accuracy

  • per fold train loss (for example, fold #55 is the 5th fold of the 5th epoch, with 10 folds in each epoch)

  • per fold train accuracy

  • per fold validation accuracy

The validation accuracy (per-fold and per-epoch) reaches close to 100% very quickly, within 9 epochs of 10-fold validation in each epoch.

I use all of the data in each fold, for the training and validation processing.

My questions are:

  • Is there something wrong with my approach?

  • Is it correct to use epochs with k-fold cross validation using all data in each fold, while training neural networks?

  • Could it be said that the weights are able to 'remember' the data in between epochs, which is why the network learns so quickly when epochs are used? In consequence, is the approach overfitting the data, given that all data in all folds are being used?

  • Instead of using full dataset, is it better to use mini-batch samples from the pool of 9-folds to train, reporting validation accuracy on the full dataset of the 10th validation fold, and then reporting average of the validation accuracy in each epoch? (and perhaps, per-fold and per-epoch training loss as well).

  • However, is the mini-batch approach over many epochs just a slower way than training using the full dataset in all training folds, and eventually lead to the same probably overfitting results?

Please see the tensorboard graphs below for the trends of these stats:

  1. Validation Accuracy per epoch (total 9 epochs, 99.78%)

enter image description here

  1. Validation accuracy per fold (reaches 100% in the 51st fold) enter image description here

  2. Train accuracy per epoch (reaches 100%) enter image description here

  3. Train accuracy per fold (reaches 100%) enter image description here

  4. Train loss per epoch: enter image description here

  5. Train Loss Per Fold: enter image description here

$\endgroup$
  • $\begingroup$ you are already mentioning overfitting a lot. One thing i realized is you wrote you are using all (100%?) of your data for training und for validating? Try to split them up, your test data shouldn't be used yet in your training. $\endgroup$ – mischva11 Jul 22 '18 at 7:38
  • $\begingroup$ Yes, I am using all the data in each fold, but folds are always separate ( 9 training folds and 1 validation fold). $\endgroup$ – Nitin Jul 22 '18 at 8:37
  • 1
    $\begingroup$ It is certainly wrong to change folds after each epoch, but maybe I didn't understand your approach. $\endgroup$ – Michael M Jul 22 '18 at 8:46
  • $\begingroup$ I’m doing 10-fold cross validation within a single epoch, using 9 folds of training data, and 1 fold of validation. This process is repeated within a single epoch, until all 10 folds are used for validation. For each fold I get a validation accuracy, and in a single epoch, I average the accuracy across the 10 validation folds. $\endgroup$ – Nitin Jul 22 '18 at 11:29
  • $\begingroup$ So do you have 10 different models throughout the process? After the first epoch, you have 10 models, built on each of the ten fold combinations. Do each of those move along to the second epoch, to be trained on the same fold combinations? $\endgroup$ – Ben Reiniger May 10 at 14:39
1
$\begingroup$

Your approach is incorrect. When you train your model, you NEVER allow your validation fold to become one of the training folds. As an example, let's say that you will have your model train for 30 epochs. You select your 9-folds that will be used for training and you select your 1-fold to be used for validation. Now train your model for all 30 epochs and DO NOT allow any of your training and test folds to interchange after any individual epoch. If you do that, your model is no longer valid. It will have seen all of the data during training, which means it will eventually get to 100% accuracy given enough epochs.

After these 30 epochs, now you select a new 9-fold combination and a new validation fold and repeat the process!

$\endgroup$
  • $\begingroup$ Thanks for the idea. It is interesting. The only thing with this approach for this dataset, is that the authors specifically recommend using k-fold cross validation. Please see this link: urbansounddataset.weebly.com/urbansound8k.html, the section: "Don't evaluate just on one split! Use 10-fold (not 5-fold) cross validation and average the scores" $\endgroup$ – Nitin May 11 at 7:13
  • $\begingroup$ But that's what @Austin suggested. The difference(?) is that you should make the fold splits consistent throughout the epochs; just make the splits up front, and for each combination of folds train the entire model (all epochs), scoring on the validation fold. $\endgroup$ – Ben Reiniger May 11 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.