# Neural network architecture that takes in matrix as input and outputs matrix?

I am trying to build a neural network that takes in matrix (read: m x n) data at a particular timestep and then outputs the prediction for the next timestep. While retaining spacial relationships. For instance, given sea surface temperatures at one month, predict the next month.

My current idea is to use a few layers of CNN, reshape into vector, feed it through fully connected layers, then reshape it back into the original m x n shape. However, I do not know if this is best practice, as it might be difficult for the network to understand the relationship between the vector and reshaping into map shape.

Additionally, I would like to feed in other inputs, such as a month index.

How should I proceed?

• Why not just output a vector and then reshape the vector to be a matrix? For example, if you have $mn$ outputs, then you can reshape that to be a $n \times m$ matrix.
– Sycorax
Feb 20, 2020 at 15:37
• So then the model would need to understand how the positioning of the elements inside the vector maps to locations on the final image output. I would think that would be difficult Feb 20, 2020 at 19:58
• Why would this be hard? If all of your output have $mn$ elements, then all you have to do is use a consistent map from output index $i$ to matrix index $(j,k)$.
– Sycorax
Feb 20, 2020 at 20:33
• I may be wrong. My understanding is that a convolutional layer can be viewed as a specialized version of a fully connected layer that places emphasis on nearby values. By mapping it into a fully connected layer, it might be sufficient, but it could be difficult for the network to learn which pixels are nearby spatially, and thus affect each other. Feb 22, 2020 at 4:24
• These networks are used all over. For an example of a CNN-FC-CNN network, take a look at any MNIST autoencoder.
– Sycorax
Feb 22, 2020 at 14:06

Neural networks in computer vision usually operate with inputs of shape (m, n)— images—so there is nothing unusual about that.