Concatenation or separate channels for a CNN

Question

let's say I am classifying time series data from multiple channels in a biomedical setup (e.g. 12 lead ECG).

I have been reading this paper on a CNN-based (ResNet) architecture for assesing the quality of a foetal ECG, where each channel is fed into a seperate network and finally concatenated to be fed into a classifier (MLP), as shown below:

My question is this, I could see the merit in doing this task in this manner with separate flows, but I am curious as to why one could not concatenate the spectrograms and have that one BIG image-like object flow through a bigger CNN? This can be done in a depth wise fashion, so the dim'n would be (height, width, channels) or make a big image. Is it simply because then the model would learn features specific to each channel in a more efficient way by having a seperate flow for each channel? then why can't we just have more filters in a bigger CNN looking at the bigger concatenated spectrograms? What more advantages are there in this kind of approach of having separate flows for each channel? My intuition about the computational resources is the bigger model would have close to the same number of parameters/Flops as the separate channel model as well, am I correct?

Answer 1

I'm answering this purely from a deep learning architecture persepective. There may well be domain-specific reasons why the authors have chosen the architecture described. This review paper: Stahlschmidt et al.'s Multimodal deep learning for biomedical data fusion: a review may provide some domain-specific insights.

Data fusion (the term for what you are describing) can be done at the input layer (early fusion), immediately before the output layer (late fusion) or somewhere in between (middle or intermediate fusion). If the fusion is followed by CNN layers, the two inputs are usually stacked channel-wise. This means that they have to be compatible - so the same sizes across all dimensions except the channels. You probably also would want the time steps to be synchronised.

When using early fusion, the model will learn joint features from the inputs; while with late fusion it learns separate features from each input. Middle fusion does a bit of both. Which one is best depends on the data.

If you keep the same convolutional parameters, and have the same total number of filters for each layer, then the number of parameters in your model will be about the same whichever strategy you use.