1
$\begingroup$

Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding spectrogram (frequency vs time plot) images. The image shown below shows an example how the whale calls look like in the Label B(B is a species of whale and C stands for negative samples) of the spectrogram.

orca detection

Since the audio files will be of varying length, the pre-processing step would involve padding all the shorter length samples with zero to have a fixed length for all files. So the spectrogram images of all those shorter samples will have the whale call in the beginning (or somewhere) with the majority of the frequency-time area as mere noise from the padding. (The above example divided the audio sample into some frames (to divide them into positive and negative classes) and labelled them as B,C.)

If we were to use the spectrogram images as such, this would hinder the generalization of our CNN model to a large extent.

Or if we save the output from the pre-processing to .npy format (binary form), I guess this could go unnoticed (or not?). What will be the consequences of saving the images in .npy format and then using in our model

I am not sure whether I am correct with my reasoning or not. Can anyone help me out?

$\endgroup$
1
$\begingroup$

A well trained classifier should be able to learn that there is no information in the padding. So it might not hurt as much as you expect.

To handle varying lengths of audio clips, and to reduce complexity of the model, the input is often split into analysis windows. These are then fixed length, say around 1 second. Only the last window then needs to get some padding, no matter if the audio clip is 2 seconds or 20. Predictions for an entire clip is then computed by voting on analysis windows predictions.

$\endgroup$
0
$\begingroup$

Firstly, you can see this pre-processing as a blessing. Since you have lots of choices of how much padding is before and after etc., you can use this as an data augmentation method.

If you do that, it may very well be that you approach could work.

Secondly, you should consider using a (Mel-)spectrogram. There may be a very good reason that's the standard approach most people use for audio. In that case you could create your features using the pre-trained VGGish model by Google. That neural network uses the spetrogram as an input to 1-D convolutions (along the time axis) with the value being the intensity at the frequency band and the different frequency bands as channels. You then build a neural network with either Google VGGish as the first few layers (only an option if you use tensorflow or convert their network to something else) or you build a new NN with their model's output as your input. You could also just use their model output as the input at different time steps for e.g. a RNN (LSTM) or some kind of transformer model with attention.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.