# Do I need 3 RGB channels for a spectrogram CNN?

I am computing a linear spectrogram of an audio signal.
https://en.wikipedia.org/wiki/Spectrogram

The spectrogram is a 2-D matrix with time on the x-axis and frequency on the y-axis. The traditional approach is to apply a color mapping to convert the spectrogram (with values from -80 to 0) to an image with three color channels. This effectively makes it a 3-D tensor. Then we feed that into the convolutional neural net.

Does this color mapping step really need to be done? The color mapping will have different nonzero ranges on different channels, but what are we gaining? The color map is just derived from the 2-D matrix so I don't see how there is any information increase, but certainly it adds to the computational complexity. And it seems like it might really depend on which color map is used, adding another layer of complexity.

Do I really need three channels since all of the information is in the 2-D matrix? How does this affect the ability of the CNN to detect certain audio markers?

• The colors come from a choice of how to map the greyscale values to a more pleasing/visible image, right? I can't see how the color is adding any additional information, because it's just a different way to display the greyscale data. – Sycorax says Reinstate Monica Sep 11 at 17:05
• @Sycorax, The spectrogram that people view are for pleasing visualizations, but when fed into the CNN it's just data. Different color maps map different intensities to different colors, so it seems like changing the colormap would impact the different channel values and maybe the effectiveness of the CNN. – Gene Sep 11 at 18:16
• That sounds like another reason to prefer the greyscale: you care about the underlying data, not the aesthetic choices that went into choosing a particular color scheme. – Sycorax says Reinstate Monica Sep 11 at 18:20

No, the color is just an aesthetic aspect of the spectrograms. A grayscale spectrogram contains all of the relevant information in its pixel intensities. You can tell because in most application you can select the colormap yourself (i.e. if you want the intensities to take shades of red, greed, blue, purple, etc.).

I can personally confirm this as I've successfully trained CNNs on grayscale spectrograms!

• Ahh, relevant experience. Thanks for confirming this! – Gene Sep 11 at 17:54
• You're welcome! :) – Djib2011 Sep 11 at 22:55

You are mapping intensity as function of time and frequency. That is: you have a map like $$\mathbb{R}_+^2 \to \mathbb{R}_+$$.

This is much like many other types of mappings from 2d coordinates to some 1d level. E.g. height maps, temperature maps, etc.

Technically, you would indeed not need colours or 3 RGB channels, to express the (1 dimensional) result.

However, beyond the aesthetic purpose of the use of colours, you may wonder about certain functional/practical purposes of the colours. (for example: easier recognition of specific features, easier recognition of the scale including when people are colour blind, etcetera).

For instance, in topographic maps you find colours: blue (below sea level), green (moderate hight, ie where grass and plants grow), grey or brown for mountains.

These colours in the height map could be left out from a technical point of view and the graph could just as well be created in greyscale.

The use of colours becomes a neccesity when you wish to plot multiple one dimensional features in a single plot (like plotting each feature by a different colour or other aspect of the spectrum). That is, when you wish to plot more than a one-dimensional result.

Such complex colourful maps are not easy to read. And as a consequence not many are made and there are not many examples.