# RNN not matching expected output

I am trying to build an RNN to predict a time-series signal based on knowledge of 3 others that I believe to be related to the output. I am using the rnn Lua Torch package, but I am quite new to the area and the platform. I admit to be struggling a bit since much of the documentation for Torch rnn appears to be for modules (such as recurrent) that have been deprecated and removed from the current version (checked out latest commit 83a5f17 dated last change Aug 23, 2017 in https://github.com/torch/rnn), and many of the examples in the repo seem too complex for my needs.

What I currently have is the following:

opt = {
rho = 10,-- maximum number of time steps for back-propagate through time (BPTT)
inputSize = 3,
outputSize = 1,
hiddenSize = 10,-- number of hidden units used at output of the recurrent layer
batchSize = 10,-- number of training samples per batch
nIterations = 2000,-- max number of training iterations
learningRate = 0.001-- learning rate
}
local rm = nn.Sequential()

rnn = nn.Sequencer(rm)
criterion = nn.SequencerCriterion(nn.MSECriterion())


And the batch training follows the rnn module examples:

minErr = 100000 -- report min error
minK = 0
avgErrs = torch.Tensor(opt.nIterations):fill(0)
for k = 1, opt.nIterations do
---- 1. create a sequence of rho time-steps-----
local inputs, targets = {}, {}
for step = 1, opt.rho do
-- batch of inputs
inputs[step] = inputs[step] or sequence.new()
inputs[step]:index(sequence, 1, offsets[step])
-- batch of targets
offsets:add(1) -- increase indices by 1
offsets[offsets:gt(opt.dataSize)] = 1
targets[step] = targets[step] or output.new()
targets[step]:index(output, 1, offsets[step])
end

---- 2. forward sequence through rnn-------------
local outputs = rnn:forward(inputs)
local err = criterion:forward(outputs, targets)

if k % 100 == 0 then
print('Iter: ' .. k .. '   Err: ' .. err)
end

avgErrs[k] = err
if avgErrs[k] < minErr then
minErr = avgErrs[k]
minK = k
end

---- 3. backward sequence through rnn -------------

----------------------------------------------------

rnn:updateParameters(opt.learningRate)
----------------------------------------------------
end


The plot below shows the thee input channels at the top, followed by the predicted output from the RNN, and the expected output at the bottom. FOr the sake of simplicity training and testing have used the data dataset, though I get similar results when I use disjoint datasets. As you can see, the expected output is quite a simple sawtooth-wave, and I intuitively expected the important phase details to emerge from the three input values and readily determine the right structure of the output, but actually the predicted output is much more like a square-wave signal.

I have tried varying the number of hidden nodes, adding a second RNN layer, ramping up the number of iterations, and varying the batch length and number of back-propagation steps, all to little avail. I wonder whether I have simply designed the RNN incorrectly. Are there some obvious things that I should try to improve things?

• rnn:forget() needs to be called at the end of each training iteration to clear out the previous state.