I am trying to build a neural network that takes in matrix (read: m x n) data at a particular timestep and then outputs the prediction for the next timestep. While retaining spacial relationships. For instance, given sea surface temperatures at one month, predict the next month.

My current idea is to use a few layers of CNN, reshape into vector, feed it through fully connected layers, then reshape it back into the original m x n shape. However, I do not know if this is best practice, as it might be difficult for the network to understand the relationship between the vector and reshaping into map shape.

Additionally, I would like to feed in other inputs, such as a month index.

How should I proceed?

  • $\begingroup$ Why not just output a vector and then reshape the vector to be a matrix? For example, if you have $mn$ outputs, then you can reshape that to be a $n \times m$ matrix. $\endgroup$
    – Sycorax
    Feb 20, 2020 at 15:37
  • $\begingroup$ So then the model would need to understand how the positioning of the elements inside the vector maps to locations on the final image output. I would think that would be difficult $\endgroup$
    – jeffhu
    Feb 20, 2020 at 19:58
  • $\begingroup$ Why would this be hard? If all of your output have $mn$ elements, then all you have to do is use a consistent map from output index $i$ to matrix index $(j,k)$. $\endgroup$
    – Sycorax
    Feb 20, 2020 at 20:33
  • $\begingroup$ I may be wrong. My understanding is that a convolutional layer can be viewed as a specialized version of a fully connected layer that places emphasis on nearby values. By mapping it into a fully connected layer, it might be sufficient, but it could be difficult for the network to learn which pixels are nearby spatially, and thus affect each other. $\endgroup$
    – jeffhu
    Feb 22, 2020 at 4:24
  • $\begingroup$ These networks are used all over. For an example of a CNN-FC-CNN network, take a look at any MNIST autoencoder. $\endgroup$
    – Sycorax
    Feb 22, 2020 at 14:06

1 Answer 1


Neural networks in computer vision usually operate with inputs of shape (m, n)— images—so there is nothing unusual about that.

I would avoid using fully connected layers in this case and go for a fully-convolutional network. Have a look at U-Net, which has become quite a standard architecture for image-to-image tasks: https://arxiv.org/abs/1505.04597

Feeding it additional data of could be tricky, but fusing them in as additional channels at the bottom part of the U shape could work. Proper scaling might be necessary.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.