# How to feed multivariate spatio-temporal data into cnn?

After trying to find an example for quite a while, I finally came to ask my question here:

What I have:
I have a temporal sequence of 2d spatial data with 100 cells(or pixels) in longitude and 30 cells latitude direction. This sequence has weekly intervals over a period of approximately 15 years, ending up in roughly 800 time steps. For each time step I have 3 channels, one of them is the numeric target variable I want to predict. For a better understanding I made the following figure: What I want to do:
I want to predict the target variable for one or more time steps ahead. For the time steps I want to predict, the two other channels are known which means I want to provide them to the model as exogenous input. I want to do a convolution over time and space (as represented by the filter in the figure above). According to my understanding, this would be a 2d convolution on 3d data (first dimension = longitude, second dimension = latitude, third dimension = time)?!?! I want the filter to be at least 52 time steps (= one year) deep as there is a signal delay/lag of up to several months between the target variable and the other channels. The task is quite analogous to feeding videos into a cnn, where I want to make a prediction for one of the rgb channels.

My question is:

• How do I need to structure the data as cnn input? Should it be as a 4d or 5d array?
• (How to implement exogenous input for prediction in keras?)

What I use:

• Keras with tensorflow
• R (preferentially) or Python

Edit: After further research and robobors answer:

I just created a drawing to clarify my understanding of right the data structure I need as input (5d tensor): Is my understanding of the input shape correct?

There are several different types of architectures you could consider to solve this problem. You chose a solution fully based on CNN's, which is reasonable, but another apparent choice would be to use something like LSTM-CNN's. For the sake of simplicity let's stay with a full CNN-based effort.

You will require 3d convolutions. Look. In this case, your array would be five dimensional, e.g. [sample_number, time_step, x, y, channel]. Yet, convolutions are performed over the three middle dimensions; time being treated just as the spatial dimensions. At this point, your case is a bit special due to the future availability of two channels, while the third one is missing.

A straight forward approach would be to extract samples of 52 timesteps. Modify then the last step by discarding* channel 1, and instead using this channel's information as the last timestep as target variable. *Discarding, most elegantly, would be some masking operation, yet masking in CNN's generally stirs discussion. Google might yield some workarounds, or see discussions like this. The common man's masking, which I would recommend as a baseline for your case, would be to just replace the channel's value by some constant (e.g. 0).

There are yet some questions remaining:

1. For how long do you have reliably the "future data" of the two channels? If you have it in advance for quite a period, you should consider utilizing more of these time steps.
2. Is data of these two other channels really this reliable? Are you sure you are allowed to provide this data, and that it makes sense from a prediction perspective?
3. Take well care when validating your approach. You generally will predict one step in advance with this baseline method. You then do one step and include the generated information to predict t+2. This requires a very conservative approach when doing a clean validation.

Good Luck.