8
$\begingroup$

I am training a neural network for Audio classification.

I trained it on the UrbanSound8K dataset (Model1), and then I wanted to evaluate how different levels of added noise to the inputs influenced prediction accuracy. Baseline accuracy Model1 = 65%

As expected, higher levels of noise resulted in lower accuracy.

Then, I decided to perform data augmentation with noise (Model2). So I took the dataset, and I duplicated it with the same files but adding pink noise (+0 dB SNR) to them.

As expected (by me), the overall accuracy increased (a very tinny bit though, 0.5%), and the network became more robust to noise corruptions of the inputs.

However! One thing that I was not expecting was that now the network has reduced its accuracy when predicting only uncorrupted-with-noise inputs (validation inputs). Somehow, it has overfitted to the clean inputs, thus reducing prediction accuracy on these audios.

So, in numbers, Model2 predicts with 69% accuracy on noisy inputs (not necessarily the same noise that was trained with), and 47% accuracy on clean inputs.

Is there any explanation or intuition into this result?

I was expecting that the network, having now more and more varied training data, would learn more meaningful features. I guess it is more difficult to overfit to the noisy inputs, but still I don't understand why it has overfitted to the clean inputs mainly.

------------------------------------------------- EDIT 1 ----------------------------------------------------------------

Another piece of information that may be helpful:

Even when evaluating Model2 on noisy inputs with very little noise, still the network performs way better than on just clean inputs (which are very much the same as the inputs-with-little-noise to our ears)

$\endgroup$
  • $\begingroup$ when you say "reduced its accuracy when predicting only the original inputs", do you mean the uncorrupted training data or the uncorrupted validation data? You should expect to worse on the former but better on the latter. $\endgroup$ – shimao May 17 '18 at 20:14
  • $\begingroup$ I meant the uncorrupted validation data @shimao $\endgroup$ – sdiabr May 18 '18 at 13:53
  • $\begingroup$ Are you normalizing your inputs before or after data augmentation? How are you normalizing? $\endgroup$ – Jonny Lomond May 23 '18 at 10:13
  • $\begingroup$ I did not perform any input normalization.. $\endgroup$ – sdiabr May 23 '18 at 10:50
  • $\begingroup$ does your training set have both 'clean' and 'noisy' data now or just 'noisy'? $\endgroup$ – Zhubarb May 23 '18 at 16:55
2
+50
$\begingroup$

You fit a model to error free input features. Then you added some error (noise) to your same data and fit the model again. You observed worse prediction on noisy inputs (inputs with error) than on noise free inputs (inputs without error). You expected the model to be just as good as previous model on noise free inputs and better on noisy inputs.

You did not add more training data you simply duplicated the same data with noise. Intuitively, a model trained on ALL noise free inputs is going to have more accurate predictions when inputs are also noise free than a model trained on ALL noisy data. Similarly, my intuition is that a model trained with all noisy data will be more accurate when predicting from noisy inputs than a model trained with all noise free data. If you have some mix of noise free and noisy data, then my intuition is you will have better predictions on noisy data than a model trained with only noise free data and better predictions on noise free data than a model trained on only noisy data. This seems consistent with what you observed.

EDIT:

Basically, overfitting occurs when we mistake noise in the data for signal. I use the term noise in th conceptual sense of useless information or information specific only to training data. If this occurs the model fits the training data well, but does not generalize well. Imagine we have points and the model interpolates all the points. If the points are noisy then this behavior is undesirable. My rudimentary knowledge of data augmentation is that it reduces overfitting because when we add noise to training data the model we fit will tend to balance the error between these nearby points in order to minmize the overall error. This model is better in an average sense in that it has less error when predicting both noisy and noise free data. It will generalize better to data that may be slightly different than training data. However, the model does not distinguish between noisy and noise free data so it has worse performance on noise free data because it mistakes some signal for noise.

$\endgroup$
  • $\begingroup$ Thanks for the answer @Nat. A few remarks: In the first paragraph you say "then you added some ... and fit the model again", I did not fit the model again, I just tested the model trained on clean inputs, on noisy inputs. What you say at the end is true, but that is not what I have observed. What I observed is that the model trained on clean_inputs + noisy_inputs predicts noisy inputs better than a model trained on clean inputs (kind of obvious) BUT! it predicts with less accuracy clean inputs. $\endgroup$ – sdiabr May 19 '18 at 7:55
  • $\begingroup$ If I understand, I think the word choice “fit the model again” is problematic. You have two models: 1) trained on clean data 2) trained on clean and noisy data. That is what I was trying to say. I don’t understand how what you describe in your comment is different than what I said in my answer. “...better predictions on noise free data than a model trained on only noisy data.” The model trained on only noisy data is a hypothetical model (3). $\endgroup$ – Nat May 19 '18 at 12:58
  • $\begingroup$ Sorry yeah, I think I misread something. But the question is: Why when I train the model on noise+clean inputs, the accuracy prediction on clean inputs decreases with respect to the accuracy prediction of clean inputs when evaluated on the model trained with only clean inputs... Isn't what I did the whole point of data augmentation? I thought that feeding more varied data to the network, will force the network to learn better features and perform better. $\endgroup$ – sdiabr May 19 '18 at 15:48
  • $\begingroup$ I edited my answer in response to your comment $\endgroup$ – Nat May 20 '18 at 3:21
  • $\begingroup$ Why do you say that "the model does not distinguish between noisy and noise free data" ? Thanks for the answers, I appreciate it. Still, I was hoping for a bit more "scientific" answer, so I'll keep the answer unaccepted for now. $\endgroup$ – sdiabr May 20 '18 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.