Suggestions for Neural Network Structure for Time-Series prediction with constant covariates

Question

I've been working on a time series prediction problem and wondered if someone has run across a similar problem structure & can make a suggestion on how to structure the training data, network, or similar to yield reasonable regression performance.

My data is structured as follows:

1. I have a time series of outputs y(t) which I am trying to predict.
2. I have a series of (constant) covariates say: {a,b,c} that contain information about the time series.
3. The time series are different lengths (is usually a function of the covariates).
4. The covariates result in non-linear effects in the outcome, which may manifest at different times, and there can be interaction effects between covariates.

An example (of made-up data) to show the general data structure is below:

t   a   b   c   y
0   17.5    3.3 0.3 0.0
1   17.5    3.3 0.3 -0.8
2   17.5    3.3 0.3 -1.0
3   17.5    3.3 0.3 -0.6
4   17.5    3.3 0.3 0.5
5   17.5    3.3 0.3 2.2
6   17.5    3.3 0.3 4.6
7   17.5    3.3 0.3 7.6
0   20.3    3.6 0.3 0.0
1   20.3    3.6 0.3 -1.1
2   20.3    3.6 0.3 -1.4
3   20.3    3.6 0.3 -0.9
4   20.3    3.6 0.3 0.4
5   20.3    3.6 0.3 2.4
0   16.0    2.6 0.3 0.0
1   16.0    2.6 0.3 -0.6
2   16.0    2.6 0.3 -0.8
3   16.0    2.6 0.3 -0.7
4   16.0    2.6 0.3 0.0
5   16.0    2.6 0.3 1.0
6   16.0    2.6 0.3 2.4
7   16.0    2.6 0.3 4.3
8   16.0    2.6 0.3 6.5
9   16.0    2.6 0.3 9.2
10  16.0    2.6 0.3 12.3
11  16.0    2.6 0.3 15.8
12  16.0    2.6 0.3 19.7

I am able to train individual models for each time-step (e.g. many models with y1=f(t=1,a,b,c), y2=f(t=2,a,b,c)), and this does generally work; however, I'm looking for an alternative to this which would allow for training, QA/QC and prediction using a single model.

In my real-world example, I actually need to predict several time series that are related, and depend on the same covariates (but different & inter-related physical processes) - e.g. y1(t), y2(t), ... yn(t)

I've been tempted to use Recurrent Neural Networks (RNN). These are generally what I'm looking for, except I'm unsure of how to feed constant covariates into the model in an effective manner.

What model types / structures should I be considering for this type of time series prediction problem?

(I'm working in either R or python, so references with respect to either of those would be particularly helpful.)

Answer 1

If you use RNNs, static features can be fed just like varying features. As noted, with most out-of-the box implementations (if any could be called that) you'll feed the same values over and over again. I think we need to be implementation- and data-specific to get a useful discussion about how this can be approached, or if it's worth approaching.

Depending on the size of your data and your level of skill this performance gain will most likely be outweighed by the time it takes to implement. It all comes down to how much much you like abstraction, which implementations you are familiar with etc.

If you are predicting multiple time series with the same step it sounds like you have a regression problem in each step. Let your network have many outputs and set the loss function as sum of squares $\textrm{loss}_t = \sum_k (y_k(t)-\hat{y}_k(t))^2$. You could also embed it into some kernel and learn the parameters in that space, like letting your network output the parameters in a gaussian.

To get useful answers I'd recommend you to give the dimensions of the data:

• How long is the longest sequence?
• How many sequences do you have?
• What's the distribution of sequence lengths?
• How large temporal dependencies do you expect to need?
• etc