0
$\begingroup$

I know RNNs (with LSTMs or GRUs) are now one of the most promising options for modelling sequential data, when ordering of the data matters. However, sometimes there are also some categorical variables associated to each data point from the sequence that also matter to model the pattern. For example, consider some time-series with daily frequency; one could have a sequence like $343, 365, 222, 34, 675, 546$ corresponding to the days-of-year $1, 2, 3, 4, 5, 6$ and another similar sequence corresponding to the days $45, 46, 47, 48, 49, 50$. For some reason, day $7$ could have high probability of yielding a value very similar to the previous ones, e.g. $388$, while day $51$ would have high probability of yielding some value 2 orders of magnitude greater, e.g. $23467$. Therefore, my question is:

How can we model sequential data together with class variables associated to each data point?

It occurred to me I could think about this as modelling more than a sequence simultaneously - one sequence being the data points (the "data sequence") and the others being the class-labels ("class sequences"), which for some use-cases could be deterministic (as in "Tuesday always comes after Monday" or "day 7 comes after day 6", etc.). I would then feed two (or more) synchronized sequences and try to predict the next time-step value for each. E.g.,

Input: [[  1,  2,  3, 4,  5,  6],  -> Output: [[  7],
        [343,365,222,34,675,546]]              [388]]

Input: [[ 45, 46, 47,48, 49, 50],  -> Output: [[   51],
        [342,368,242,43,612,549]]              [23467]]

Is this the correct approach? How should one model an RNN with several inputs? In particular, what's the input and what's the output? Should one use one-hot encoding with a high dimensional vector? No, right?

Another case where I think this could be applied would be trying to model many correlated "data sequences" simultaneously (e.g. temperature time-series for several cities in a country)

Also, how can we implement something like this using Tensorflow?

$\endgroup$
1
$\begingroup$

The input, as you suggested, should be the sequence you're trying to predict concatenated with the day of year. The output should just be the predicted value of the sequence at the next time-step, there's no need to predict the next day of year as well, since you already know what it will be.

One hot should be ok with just 365 categories. It only starts to get problematic with language models when you have tens of thousands of words in your vocabulary.

Another thing which may be important is to apply some sort of transformation to your sequence (maybe log) which will make predicting huge values much easier without having huge gradients and associated issues.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.