0
$\begingroup$

So I've been working on this audio-rec task for a while now, and I've had some good luck using 2D convolutions on the spectrogram of audio (I've also tried Mel-spectrograms, the difference is minor in my opinion). Up until now I've been using this network structure:

    X = Conv2D(filters=64, kernel_size=5, padding='same', activation='relu')(X_input)
    X = Conv2D(filters=64, kernel_size=5, padding='same', activation='relu')(X)
    X = MaxPooling2D()(X)

    X = Conv2D(filters=128, kernel_size=5, padding='same', activation='relu')(X)
    X = Conv2D(filters=128, kernel_size=5, padding='same', activation='relu')(X)
    X = MaxPooling2D()(X)

    X = Conv2D(filters=256, kernel_size=3, padding='same', activation='relu')(X)
    X = Conv2D(filters=256, kernel_size=3, padding='same', activation='relu')(X)
    X = MaxPooling2D()(X)

This works well, but its not awesome. Now I was thinking about why it's not really that good last night and I thought of one reason: the conv2D Kernel size! So I have a few questions to ask the StackExchange community:

1) Since audio in a spectrogram is very disperse spatially, I think my Kernel size should be much larger (keep in mind my input size is 64x64). This way the model will be able to learn more about larger segments of images, does that make sense? Additionally, should I even be using a square kernel? Time and frequency are different units, unlike images where both axes are of the same units.

2) what about Conv2D stride, and the MaxPooling2D pool size? so many hyperparameters! I'm thinking that stride might help because audio is really non-spatial in a spectrogram, I mean check this one out: enter image description here If you can see there are segments of the picture where it seems like there are lines above and below the primary (brightest) spots. This is very common in spectrograms and I believe these are actually octaves above and below the actual sound: Please correct me if I'm wrong.

Here are some details about the audio being analyzed:

length: < 0.05 seconds (yeah, that short)

num_classes: 7

desc: beatboxing audio

If anyone has any comments at all about my network structure, hyperparams, questions, insults, anything, please leave an answer or comment, and I will get back to you within a day. I'm super excited to hear what you guys think!

Thanks.

SIDE NOTE PLEASE READ: - if you have enough rep to edit this post could you please add audio and spectrogram to the tags. I don't have enough rep to create tags and I think this would help those stuck on similar challenges to find this question. Thanks!

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Thanks for the edit @Michael, much appreciated! $\endgroup$ – Nikita Jerschow Dec 22 '19 at 2:37
  • $\begingroup$ Re: parallel lines in spectrograms - yeah, even if you had a single-note melody on a single acoustic instrument, you're going to see at least some faint ones - these are overtones/harmonics. It's actually what makes the exact same notes sound different on different instruments. $\endgroup$ – jkm Dec 24 '19 at 9:13
  • $\begingroup$ As for Conv filter sizes - covering larger areas can be handled simply by the fact that deeper layers operate on the pooled outputs of the previous ones, e.g. a 3x3 convolution of pooled 3x3 convolutions effectively covers a 9x9 area. Larger individual filters mostly boost the 'sophistication' of the local patterns you could detect, but it's a fairly computationally expensive hyperparameter. $\endgroup$ – jkm Dec 24 '19 at 9:24
  • $\begingroup$ You say that your audio is 0.05 seconds long, what do you mean by that? What are the dimensions of your spectrogram patches that your network processes? And what is the time and frequency resolution of the spectrograms. The appropriate kernel sizes depends a lot on this! $\endgroup$ – jonnor May 16 '20 at 13:51
  • $\begingroup$ Also, how many samples is your training set. The suggested network has a very large amount of parameters, and will require a lot of samples to train. $\endgroup$ – jonnor May 16 '20 at 13:52
1
$\begingroup$

Convolutional Neural Networks (such as the one you utilize) tend to increase the information density per neuron from the lower layers to the higher ones. Therefore, it is useful to decrease the kernel size when reaching deeper layers as you already did. From experience, I can tell you that $5\times 5$ kernels in early layers with $3 \times 3$ kernels in subsequent layers are a very good starting point.

Also it is usual to put a Dense layer as the last layer of your network (with Softmax activation) since arrangement of the labels in the last layer should be independent of the spatial information that is carried by the CNN layers.

My question now is: why don't you use more advanced features such as MFCCs, PCEN (https://arxiv.org/pdf/1607.05666.pdf), Constant-Q-Transform (for music mostly), or Modulation Domain approaches (https://pdfs.semanticscholar.org/62d6/954ddf58aa9420bbe59146f7e632f4df01b1.pdf) that have been shown to achieve better results in general on audio tasks?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the answer! So, if I wanted to capture as much spatial information as possible, would I ever use convolutional strides? and what about hyperparameters for max pooling layers? I'm gonna check out these links and get back to you with results. $\endgroup$ – Nikita Jerschow Dec 29 '19 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.