2
$\begingroup$

I am working with a time-series, and believe that time in seconds since start is a useful feature. I am however not sure how to best input this to my recurrent neural network. I don't think the naive approach of just entering the naive value is the best way, and I also don't think that letting the RNN keep track of it on its own is good either.

What has been done before, and what considerations should I keep in mind if I want to use time as an input variable?

Clarification

Let's say that at each time step, t (seconds as integers, ~0-1800), I have a feature vector x. I believe knowing the value of t is useful i.e. how much time has elapsed since the start of a process. How do I best input t to the network? I.e. is there something better than concat(x, t)?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Which are your reasons to think that just entering the value is not the best way? $\endgroup$
    – Winkelried
    Commented Apr 8, 2018 at 11:17
  • $\begingroup$ The range of values is quite large, roughly $\in [0, 1800]$, and from my experience that is not very nice for a neural network. This is somewhat alleviated by batch normalization, I was just wondering if there were some best practices. $\endgroup$
    – Faur
    Commented Apr 11, 2018 at 17:30
  • $\begingroup$ Would scaling solve the range problem? $\endgroup$
    – Winkelried
    Commented Apr 12, 2018 at 14:35
  • $\begingroup$ Scaling how? Just $t'=t/1800$? or do you mean log scaling? Linear scaling is redundant because of batch norm. I don't know if log scaling is smart $\endgroup$
    – Faur
    Commented Apr 15, 2018 at 13:07
  • $\begingroup$ Doesn't the fact that linear scaling is redundant imply that the large range of values wouldn't be a problem? $\endgroup$
    – Winkelried
    Commented Apr 15, 2018 at 19:05

1 Answer 1

Reset to default
1
$\begingroup$

You are essentially applying a recurrent neural network to solve a Markov problem. The neural network can be trained to learn a so-called hidden Markov model, in your time data. You try to predict the probabilities of the possible next states at time $t$, given the current state $t-1$, the previous states (represented by feature variables), and state information: $t-2$, $t-3$, $\ldots$.

One first question you need to address, is whether the state at $t-2$ has any extra predictive power as to which next state is being reached. As the underlying mechanism is assumed to be unknown, you ought to experiment with this - adding or retracting historic, previous state information from your recurrent neural network.

There are basically two alternative strategies to follow: They both involve training the RNN to convolve along the time/state axis. According to one strategy, you ridirect the output vector of the RNN to special input units, which exist alongside the regular feature-input nodes. According to the other strategy, you provide the vector of hidden-node values to the special input units. The implicit assumption of both these approaches is that you are dealing with a first-order Markov chain. State-information before $t-1$ is completely incorporated into the current state.

Keep a subset of your data as a separate test set.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think you misunderstood what I meant. I have added a clarification $\endgroup$
    – Faur
    Commented Apr 11, 2018 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.