8
$\begingroup$

I have coded up a simple real-value regression RNN in theano.

  1. What kind of dataset should I test it on?
  2. How should I go about testing it?

My structure is:

  • Univariate (for now) timeseries, $x_{in}(t)$
  • $n_{Input Nodes}$ separated by ~equal timesteps, $t_{step}$. Where, $n_{Input Nodes}$ should be sufficiently large to capture a recurrence in the data
  • $n_{Hidden Nodes} = n_{Input Nodes}$
  • A prediction time lag following the final Input Node of, $lt_{step}$, where $l$ is an integer
  • One Output Node taken from the final hidden node, giving a prediction, at $t_{p}=t+lt_{step}$
  • $x_{p}(t_p)$ is the prediction of $x_{in}(t_p)$ in training data
  • Error by R.M.S.E. $\sqrt{\left(x_{in}(t_p)-x_{p}(t_p)\right)^2}$
  • Finally, each node in the hidden layer feeds through to the weight at the next timestep

Tested $y=sin(t) + 0.2*\epsilon$, where $\epsilon \sim N(0,1)$, in a sliding window. I used a historic lag of 5 data points, $y(t-5, t-4, ... , t)$, and tried to predict the following point in the curve, $y(t+1)$.

I only used 100 noisey versions of $sin(x)$ over 100 epochs for training. Results weren't too bad...

img

Thanks for the help. Code seems bug free so I'll optimise for GPU & mini-batches and ramp it up with more up to date algorithms.

Improve this question
$\endgroup$
5
  • $\begingroup$ A bit question before our collaboration, should be:... which kind of data or process are you modelling? and are you available to obtain your own desired data from the process - i.e. impulse or step or random responses?.... $\endgroup$
    – Brethlosze
    Commented May 29, 2015 at 4:37
  • $\begingroup$ On how much degree you wish to overtrain or said on other way: it is for a homework, for a thesiswork, for a real application? $\endgroup$
    – Brethlosze
    Commented May 29, 2015 at 4:48
  • $\begingroup$ Thanks for the offer of help! I'm doing this for self learning to prove that I am competent enough at programming to undertake a research project in time-series ML. I basically want to check that I haven't made any errors in my code so I want a time series that is easy to model for debugging purposes $\endgroup$
    – Alexander McFarlane
    Commented May 29, 2015 at 9:56
  • $\begingroup$ That's very nice - I have a question though, is your RNN structure, such that you take 5 inputs, and have only ONE output though, right? Thanks. $\endgroup$
    – Spacey
    Commented Jan 19, 2017 at 5:23
  • $\begingroup$ yes, correct. I suspect that is why it tries to revert direction towards zero with increasing frequency as it approaches the apex of each curve. $\endgroup$
    – Alexander McFarlane
    Commented Jan 19, 2017 at 11:23

2 Answers 2

Reset to default
6
$\begingroup$

A very simple time series to validate the correctness of your code is the one caused by the function sin(x). It's periodic nature makes it a good test function imo. Just print out (or plot) the output activations of your network and compare it with the desired values to see the performance.

Alternatively you can just test XOR like Elman did in his original paper:

101 000 011 110 101 ...

Improve this answer
$\endgroup$
5
  • $\begingroup$ Looks good, some oscillations are always to be expected unless you use a handful of tricks to tame the gradient update $\endgroup$
    – runDOSrun
    Commented May 31, 2015 at 14:40
  • $\begingroup$ seems to get quite noisey at $\pm 1$ but hopefully that'll clear up when I improve the code $\endgroup$
    – Alexander McFarlane
    Commented May 31, 2015 at 15:05
  • 1
    $\begingroup$ Might be a result of adding noise to the data. If you want to keep epochs low, you can also try RProp, it might make get you quicker convergence. But yeah, that's one of many possible optimizations depending on your goal. Good luck! $\endgroup$
    – runDOSrun
    Commented May 31, 2015 at 15:49
  • $\begingroup$ I think it might be the way I'm training. I am training by just showing the network 96 samples of 5 time steps from the function above to predict the 6th time step. So I am really trying to encode a lot of information with not that many nodes. I found that if I increase the historic lag from 5 to 15 I get vastly improved performance. $\endgroup$
    – Alexander McFarlane
    Commented Jun 6, 2015 at 6:30
  • $\begingroup$ Makes sense. Explains the errors at min and max input values - it doesnt know whether he is going up or down the sin curve without enough time context. $\endgroup$
    – runDOSrun
    Commented Jun 6, 2015 at 8:04
1
$\begingroup$

There's a good list of tests in Hochreiter's paper here. Also check this and the next slide on Schmidhuber's presentation.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Welcome to Cross Validated! We are trying to build a permanent repository of high-quality statistical information in the form of questions & answers. We try to avoid link-only answers which are subject to link-rot and may be deleted. As such this is more of a comment than an answer in its own right. If you're able, could you expand it, for example by giving a summary of the relevant information at the link, or by quoting the relevant parts. Alternatively, we can convert it into a comment for you. $\endgroup$
    – Glen_b
    Commented Mar 25, 2016 at 11:01
  • $\begingroup$ Those are rather benchmarks than sanity checks. $\endgroup$
    – runDOSrun
    Commented Apr 26, 2016 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.