# How to handle 2 input representations ("channels") of a data that is associated with the exact same label?

I have a data sets of matrices with non negative values. For each learning instant I have 2 representations of the input data which corresponds to the same label.

Namely for each matrix label $\mathbf{Y}_{N\times N}$ I have 2 labels $\mathbf{X}_{N \times N}^1$ and $\mathbf{X}_{N \times N}^2$.

The output/label matrix is matrix of 0's and 1's.

Since $\mathbf{X}^1$ and $\mathbf{X}^2$ are matrices in different representations of the input signal I would like to use both as inputs. The thing is that the average value of $\mathbf{X}^2$ is significantly larger then $\mathbf{X}^1$, so I am not sure that feeding a Neural Network with 2 channels is a good idea.

1. Do you have a suggestion what should I do, beside training 2 separate networks and having some voting mechanism?
2. Can I use some multichannel network, although the scale of each input matrix is different.
3. What if $\mathbf{X}^1$ $\mathbf{X}^2$ are quite sparse (~20% aren't non zero elements)?
• How about normalizing/scaling the X1 and X2 matrices and then feeding a concatenated vector for each record as input? Also, the dimensions of Y are NxN, is that correct? Assuming a multilabel classification, the second dimensions should ideally be << N, right? Aug 28, 2018 at 5:18
• @hssay, thanks. Yes, the network is a feature detection and puts one if the feature exist in this pixel. I am not sure concatenation will be a good idea. Think about a neural network to detect edges for example. I am not sure what do you mean by "Assuming a multilabel classification, the second dimensions should ideally be << N, right?"
– 0x90
Aug 28, 2018 at 5:43
• My question is, are your sure about the dimensions of the Y matrix (NxN)? Can you add explanation to the question as to why it is NxN? IMO, it should be NxM where M << N, M is number of output categories. Aug 28, 2018 at 5:55
• @hssay, I'll explain. The network receive a NxN image and try to find the pixels in the NxN image that have the unique feature.
– 0x90
Aug 28, 2018 at 6:00

Since we' are talking about images I would assume that each image shows the same but in a different representation. Therefore, it makes sense to feed the input as one image with different channels. In this way the network is able to find corresponding features based on locality in your picture.

So if we disregard batch_dimension your input shape will look like this: (N, N, 2).

Can I use some multichannel network, although the scale of each input matrix is different.

Yes although I think the upper approach would be the better one. If you are worried about the scale of your input then just rescale it.

What if X1 X2 are quite sparse (~20% aren't non zero elements)?

Since we're talking about images, I think Convolutional Layers should be a key part of your Network. Those layers are using weight sharing, which makes them very robust against sparse inputs. Have a look at the MNIST Dataset, which is often used as a basic example for Convolutional Networks. The images there are quite sparse too.

• My output layer is NxN
– 0x90
Aug 30, 2018 at 10:58
• Yes I know and I don't see a problem with that using my proposed answer. If you have doubts in some points can you elaborate on that? Aug 30, 2018 at 11:45