6
$\begingroup$

I have coded up a simple real-value regression RNN in theano.

  1. What kind of dataset should I test it on?
  2. How should I go about testing it?

My structure is:

  • Univariate (for now) timeseries, $x_{in}(t)$
  • $n_{Input Nodes}$ separated by ~equal timesteps, $t_{step}$. Where, $n_{Input Nodes}$ should be sufficiently large to capture a recurrence in the data
  • $n_{Hidden Nodes} = n_{Input Nodes}$
  • A prediction time lag following the final Input Node of, $lt_{step}$, where $l$ is an integer
  • One Output Node taken from the final hidden node, giving a prediction, at $t_{p}=t+lt_{step}$
  • $x_{p}(t_p)$ is the prediction of $x_{in}(t_p)$ in training data
  • Error by R.M.S.E. $\sqrt{\left(x_{in}(t_p)-x_{p}(t_p)\right)^2}$
  • Finally, each node in the hidden layer feeds through to the weight at the next timestep

Tested $y=sin(t) + 0.2*\epsilon$, where $\epsilon \sim N(0,1)$, in a sliding window. I used a historic lag of 5 data points, $y(t-5, t-4, ... , t)$, and tried to predict the following point in the curve, $y(t+1)$.

I only used 100 noisey versions of $sin(x)$ over 100 epochs for training. Results weren't too bad...

img

Thanks for the help. Code seems bug free so I'll optimise for GPU & mini-batches and ramp it up with more up to date algorithms.

$\endgroup$
  • $\begingroup$ A bit question before our collaboration, should be:... which kind of data or process are you modelling? and are you available to obtain your own desired data from the process - i.e. impulse or step or random responses?.... $\endgroup$ – Brethlosze May 29 '15 at 4:37
  • $\begingroup$ On how much degree you wish to overtrain or said on other way: it is for a homework, for a thesiswork, for a real application? $\endgroup$ – Brethlosze May 29 '15 at 4:48
  • $\begingroup$ Thanks for the offer of help! I'm doing this for self learning to prove that I am competent enough at programming to undertake a research project in time-series ML. I basically want to check that I haven't made any errors in my code so I want a time series that is easy to model for debugging purposes $\endgroup$ – Alexander McFarlane May 29 '15 at 9:56
  • $\begingroup$ That's very nice - I have a question though, is your RNN structure, such that you take 5 inputs, and have only ONE output though, right? Thanks. $\endgroup$ – Spacey Jan 19 '17 at 5:23
  • $\begingroup$ yes, correct. I suspect that is why it tries to revert direction towards zero with increasing frequency as it approaches the apex of each curve. $\endgroup$ – Alexander McFarlane Jan 19 '17 at 11:23
5
$\begingroup$

A very simple time series to validate the correctness of your code is the one caused by the function sin(x). It's periodic nature makes it a good test function imo. Just print out (or plot) the output activations of your network and compare it with the desired values to see the performance.

Alternatively you can just test XOR like Elman did in his original paper:

101 000 011 110 101 ...

$\endgroup$
  • $\begingroup$ Looks good, some oscillations are always to be expected unless you use a handful of tricks to tame the gradient update $\endgroup$ – runDOSrun May 31 '15 at 14:40
  • $\begingroup$ seems to get quite noisey at $\pm 1$ but hopefully that'll clear up when I improve the code $\endgroup$ – Alexander McFarlane May 31 '15 at 15:05
  • 1
    $\begingroup$ Might be a result of adding noise to the data. If you want to keep epochs low, you can also try RProp, it might make get you quicker convergence. But yeah, that's one of many possible optimizations depending on your goal. Good luck! $\endgroup$ – runDOSrun May 31 '15 at 15:49
  • $\begingroup$ I think it might be the way I'm training. I am training by just showing the network 96 samples of 5 time steps from the function above to predict the 6th time step. So I am really trying to encode a lot of information with not that many nodes. I found that if I increase the historic lag from 5 to 15 I get vastly improved performance. $\endgroup$ – Alexander McFarlane Jun 6 '15 at 6:30
  • $\begingroup$ Makes sense. Explains the errors at min and max input values - it doesnt know whether he is going up or down the sin curve without enough time context. $\endgroup$ – runDOSrun Jun 6 '15 at 8:04
1
$\begingroup$

There's a good list of tests in Hochreiter's paper here. Also check this and the next slide on Schmidhuber's presentation.

$\endgroup$
  • 1
    $\begingroup$ Welcome to Cross Validated! We are trying to build a permanent repository of high-quality statistical information in the form of questions & answers. We try to avoid link-only answers which are subject to link-rot and may be deleted. As such this is more of a comment than an answer in its own right. If you're able, could you expand it, for example by giving a summary of the relevant information at the link, or by quoting the relevant parts. Alternatively, we can convert it into a comment for you. $\endgroup$ – Glen_b Mar 25 '16 at 11:01
  • $\begingroup$ Those are rather benchmarks than sanity checks. $\endgroup$ – runDOSrun Apr 26 '16 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.