RNN not matching expected output

Question

I am trying to build an RNN to predict a time-series signal based on knowledge of 3 others that I believe to be related to the output. I am using the rnn Lua Torch package, but I am quite new to the area and the platform. I admit to be struggling a bit since much of the documentation for Torch rnn appears to be for modules (such as recurrent) that have been deprecated and removed from the current version (checked out latest commit 83a5f17 dated last change Aug 23, 2017 in https://github.com/torch/rnn), and many of the examples in the repo seem too complex for my needs.

What I currently have is the following:

opt = {
 rho = 10,-- maximum number of time steps for back-propagate through time (BPTT)
 inputSize = 3,
 outputSize = 1,
 hiddenSize = 10,-- number of hidden units used at output of the recurrent layer
 batchSize = 10,-- number of training samples per batch
 nIterations = 2000,-- max number of training iterations
 learningRate = 0.001-- learning rate 
}
local rm = nn.Sequential()
   :add(nn.LinearRNN(opt.inputSize, opt.hiddenSize))
   :add(nn.Linear(opt.hiddenSize, opt.outputSize))

rnn = nn.Sequencer(rm)
criterion = nn.SequencerCriterion(nn.MSECriterion())

And the batch training follows the rnn module examples:

minErr = 100000 -- report min error
minK = 0
avgErrs = torch.Tensor(opt.nIterations):fill(0)
for k = 1, opt.nIterations do 
    ---- 1. create a sequence of rho time-steps-----
    local inputs, targets = {}, {}
    for step = 1, opt.rho do
        -- batch of inputs
      inputs[step] = inputs[step] or sequence.new()
        inputs[step]:index(sequence, 1, offsets[step])
      -- batch of targets
      offsets:add(1) -- increase indices by 1
      offsets[offsets:gt(opt.dataSize)] = 1
      targets[step] = targets[step] or output.new()
      targets[step]:index(output, 1, offsets[step])
    end

---- 2. forward sequence through rnn-------------
  local outputs = rnn:forward(inputs)
  local err = criterion:forward(outputs, targets)

  -- report errors
  if k % 100 == 0 then
      print('Iter: ' .. k .. '   Err: ' .. err)   
  end

  avgErrs[k] = err
  if avgErrs[k] < minErr then
     minErr = avgErrs[k]
     minK = k
  end

---- 3. backward sequence through rnn -------------

  rnn:zeroGradParameters()

  local gradOutputs = criterion:backward(outputs, targets)
  local gradInputs = rnn:backward(inputs, gradOutputs)
----------------------------------------------------

----- 4. updates parameters-------------------------
  rnn:updateParameters(opt.learningRate)
----------------------------------------------------
end

The plot below shows the thee input channels at the top, followed by the predicted output from the RNN, and the expected output at the bottom. FOr the sake of simplicity training and testing have used the data dataset, though I get similar results when I use disjoint datasets. As you can see, the expected output is quite a simple sawtooth-wave, and I intuitively expected the important phase details to emerge from the three input values and readily determine the right structure of the output, but actually the predicted output is much more like a square-wave signal.

I have tried varying the number of hidden nodes, adding a second RNN layer, ramping up the number of iterations, and varying the batch length and number of back-propagation steps, all to little avail. I wonder whether I have simply designed the RNN incorrectly. Are there some obvious things that I should try to improve things?

Answer 1

The RNN architecture seems to be fine. I eventually achieved better results with the following changes:

• rnn:forget() needs to be called at the end of each training iteration to clear out the previous state.
• try many more iterations - reasonable convergence required several 10s of thousands.
• the size of the hidden state really matters and it's worth playing around with the size and seeing how quickly the error falls in each case. In this particular circumstance about 30 nodes appears to be close to the sweet-spot.