11
$\begingroup$

I've seen a few publications that feed an RGB image of a spectrogram to a neural net, and someone claiming a network does better with RGB than grayscale or raw spectrogram.

A spectrogram is fundamentally a 2D representation with each point being a non-negative real value. Converting it to RGB adds no information. Worse, it introduces a dependence on choice of colormap, which is just noise${}^{1}$. It's worse than making grayscale images RGB, as it breaks a spectrogram's spatial dependencies by splitting into channels.

Why would a spectrogram saved as RGB ever outperform a raw spectrogram?


Clarification: originally I didn't realize this, but "RGB image" implies "image", meaning it involves a conversion step that compresses and reshapes the raw spectrogram. Additionally, it's not just any RGB in sense of $\text{R} \approx \text{G} \approx \text{B}$, but a color mapping for intensity heatmaps, like turbo (plt.imshow(np.arange(9)[None], cmap='turbo')): plt.imshow(np.arange(9)[None], cmap='turbo').

It's possible the image doesn't reshape, in which case there's no compression, but even if a colormap isn't specified, doesn't mean there's no color mapping: what matters is how the array values compare between the raw spectrogram and what's decoded as image as input to NN.

1: that was my impression at the time, it's the case with $R \approx G \approx B$, but otherwise definitely not. Depending on colormap, it can be noise (or worse) though.


Example pub with good results, but there's reason to suspect incompetence per e.g. "[1356x1071] images were lossless scaled to 32x32", which is impossible. There's no comparison with grayscale approach so we can't tell if it outperformed.


There are some "trivial" explanations I'll list to avoid answers containing them:

  1. Transfer learning: using nets pretrained on RGB
  2. Architectures tailored specifically to maximize RGB utility

While they're valid explanations, it's no evidence that RGB is any better.

$\endgroup$
16
  • $\begingroup$ Can you edit your post to share citations to these publications? $\endgroup$
    – Sycorax
    Jan 2, 2022 at 21:30
  • 3
    $\begingroup$ I suspect the reason may be that they were using deep neural networks with pre-trained feature extraction layers that had been pre-trained on RGB images? Caveat: I am certainly not an expert on deep neural networks, better on shallow ones! $\endgroup$ Jan 2, 2022 at 22:09
  • 1
    $\begingroup$ @DikranMarsupial Yes, I suspect this as a "trivial" explanation - I'll make a short list. The linked pub doesn't mention pretraining though. $\endgroup$ Jan 2, 2022 at 22:17
  • 1
    $\begingroup$ It is my understanding that often when someone says they used a particular architecture they often mean that they used that architecture pre-trained on some database (e.g. imagenet) as training it from scratch would be computationally extremely expensive, whereas it can be fine-tuned (transfer learning?) for some particular task fairly cheaply. However, as I said this isn't really my area, just read a few books. $\endgroup$ Jan 2, 2022 at 22:40
  • 1
    $\begingroup$ "saved as RGB" this might lead to confusion. I believe that your idea with this is to apply some sort of additional colour mapping when the raw data is saved as RGB. But in general, 'saving as RGB' just means to split the gray-scale layer into three red, green and blue layers. $\endgroup$ May 13, 2023 at 14:42

3 Answers 3

2
$\begingroup$

A less trivial explanation can be that converting gray-scale to RGB is effectively adding a layer of ReLU neurons with fixed parameters.

For example converting an image to RGB using the viridis colour-map is using something similar to three piecewise linear functions that can be composed out of ReLU functions.

example of applying conversion similar to viridis colour map

This addition has the effect of increasing the depth (extra layer) and width (potential extra neurons in subsequent layers) of the neural network. Both effects can potentially improve the performance of the model (if it's current depth and/or width was not sufficient).


Width

A simple example is converting a single grayscale channel to three rgb channels by simply copying the image three times.

This can be effectively like performing some ensemble learning.

Your neural network or decision tree may converge to different patterns on the different channels which can be later on merged in an average with a final layer or classification boundary.

You could also see it alternatively as effectively making several of the hidden layers three times wider (but not fully connecting them, and adding only three times more connections). This can create some potential for different training and convergence which is potentially better.

Depth

The additional color mapping layer may allow to create patterns that are not possible with less connections. The flexibility is increased.

The simplest example is an image of a single pixel that passes through a layer with a single neuron with a step function (so this is an example where even the number of neurons remain the same and the width of the subsequent network is not changed).

  • For BW, this is a two parameter function (weight $w_1$ and bias $b$) that effectively makes a classification based on whether or not the input is above or below some level.
  • For RGB, then we get two additional parameters, $w_2$ and $w_3$, for the extra channels, and this makes it possible to create more patterns. For example we can make a classification when the grayscale pixel has either a high or either low value.

Obviously one can achieve the same when not converting to rgb, and instead add more neurons or an additional layer.

  • But possibly the cases where the rgb performed better did not test this out.

  • Also the conversion to rgb, using some useful scale, is making a hardcoded seperation into shadows, middle tones and highlights, which a NN needs training and extra neurons for.

    (So in a way it is adding an extra layer which is regularised. And also it is adding pre-trained information because the human decision to choose a particular colour map instead of another; ie the human chooses the trigger points of the ReLU layer and the conversion to rgb is additional information).

Anyway, this simple example is a case where it is possible to prove that rgb can perform better (if we compare with a limited model, like only a fixed number of neurons and layers).

example of 1 pixel 1 neuron model

$\endgroup$
20
  • $\begingroup$ I don't see how this addresses RGB. All of this can be said of breaking up a 0-1 normed spectrogram into 0-to-0.33, 0.33-to-0.66, 0.66-to-1, which can be said of any 0-1 normed input, which can be said of any unnormed input replacing 0 and 1 with min and max. $\endgroup$ May 7, 2023 at 12:45
  • $\begingroup$ @OverLordGoldDragon if you are converting gray scale to rgb, isn't this just the same as feeding multiple copies of the same image to the algorithm? An algorithm which will now need three times more nodes in the first layers. $\endgroup$ May 7, 2023 at 15:51
  • $\begingroup$ @OverLordGoldDragon it can be more general and what I said may also apply to other manipulations that increase the size of the input data, without adding information. The fact that what I said here applies also more general, doesn't mean that it doesn't address reasons why conversion of gray-scale to RGB can change (and possibly improve) performance of machine learning algorithms. $\endgroup$ May 7, 2023 at 16:03
  • $\begingroup$ "same as feeding multiple copies" that's far from the case. A red ball won't appear in G and B. Conversion is also strongly colormap-dependent, which is why there's fuss over jet. My point is that as written, your explanation is generic, while both the "RGB" and "save as image" are very specific. $\endgroup$ May 7, 2023 at 16:07
  • $\begingroup$ As for increasing model size, that can be done just by increasing model size, or (as you said) duplicating the input, meaning RGB/saving is nothing special. So your description may be technically correct and a fair side-point, but missing the root of the phenomenon. But that's the best case, as it assumes a non-detrimental colormap and limited image conversion losses, and both are big assumptions, especially latter for timeseries, where the time axis may be compressed unsafely by a thousandfold. $\endgroup$ May 7, 2023 at 16:08
0
$\begingroup$

I do not have very "hard" evidence, but I have a publication under review where we have trained ResNet50 to regress some values from noisy spectrograms.

  • Pretraining in ImageNet is better than starting from random initialization
  • For pretrained networks, using color spectrograms is better than grayscale spectrograms (normalized to 0-1)

All that I have is comparative experiments in a couple of datasets, so take it or leave it :)

$\endgroup$
0
$\begingroup$
  1. Colormapping is nonlinear filtering. A color map is simply a transform; the breakup into three dimensions further interprets it as filtering and decomposition. turbo is preferable to jet for inspection (1 -- 2 -- 3) - which is to say, it's not arbitrary, and the human visual system favors it. In turbo (or jet), as one use case, we can quickly skim an image for peaks, which will be red, and we may wish to focus only on those - that's identical to the "R" channel.

  2. "Image" involves efficient (and nonlinear) compression. The standard approach to STFT compression is direct subsampling (i.e. hop_size), which aliases. An improvement is decimation, i.e. lowpass filtering + subsampling, which is a linear compression. If something so simple was effective, there'd be no need for all the sophistication of JPEG. In ML terms, we can view "save as JPEG" as a highly effective autoencoder, also effective dimensionality reduction.

There's more to say but I'll just share the main points for now.

Note that this is completely separate from using image-excelling NNs on STFT images. That can be detrimental.

Also, @Ghostpunk's answer is mistaken and misleading, as I commented. It may be owed to the popular "windowed Fourier transform" interpretation of STFT. Spectrogram losses can also be measured. Relevant posts:

Note

I realized the question, and my answer, are ill-suited for this network, and I may not be developing my answer further here. If I develop is elsewhere, I'll link it. In the meantime, refer to my discussion with @SextusEmpiricus.

Still self-accepting since, though elaboration is due, my answer can be understood with the right (mainly signal processing + feature engineering) background, and I believe it contains the most pertinent explanation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.