Time-series with RNN - how to deal with attributes that span entire sequences?

Question

I am currently trying to train recurrent neural networks for time-series forecasting, and I'm having trouble figuring out how to properly deal with attributes that stay constant over each series.

For instance, let's say I am trying to learn, for a person born between 1950 and 1970, the 30-element sequence of their yearly income between the years 1985 and 2014. Let's also suppose the attributes I have access to are:

• for some of them, relative to the person and invariant over time: their year of birth (can indicate how early they started to work), the net value of their parents they year they were born (can indicate how much help they got in their education and professional start)

• for the others, fluctuating every year: the average income in their living area, the inflation rate

In my actual problem, I am trying to predict sequences of 128 elements based on 3 attributes that are constant over each sequence and 10 attributes that actually vary over time. In particular, just like in the example I gave, the static attributes seem to be of direct importance (at least intuitively), while the varying attributes' impact is less clear.

My initial thought was to treat all the attributes the same and just train the RNN with sequences of 13-element vectors, 3 of which being static. I actually get quite satisfying predictions on my problem, but is it really good practice to mix static and varying parameters together?

Another idea I got was to construct new learning parameters combining the static and varying attributes, however I couldn't think of a function that would make sense. I guess my second question would be: should I, and how can I build attributes whose physical interpretation is not clear but would still help the time-series forecasting?

Thank you in advance for your help,

Answer 1

You should probably add in your static attributes AFTER the sequence is processed.
After you process your sequence, you have some hidden state vector that connects to however many outputs your network has. Just add on to the end of that hidden state vector however many static attributes you have and then calculate an output.
It could be a problem though if you have that data only on some people and not others. Just throwing in a 0 where the data isn't available seems like it could potentially give weird results.