1
$\begingroup$

In the spirit of the answer from maple on this thread:

Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I created some simple sine wave data to fit with a LSTM. It worked well!

However, when I did the same thing with a regular NN it also worked well?

In each case I sampled the sine wave data and used the previous sample to predict the next. So there was only one input and output each time.

LSTM's are supposed to have the edge in dealing with dynamic problems but in practice for the sine wave signal at least the results were very similar?

Can anyone point me to a problem that can be solved by an LSTM but not by a regular NN? Ideally it should be a time series problem (with numeric data).The simpler the problem the better.

If there is code (ideally Matlab) to illustrate the problem even better!!!

Thanks

Baz

t=[0:0.1:500];
A=1;
f=1;
y=A*sin(f*t);
plot(t,y)
y(end)=[];
y=y';

inputs = y(1:end-1,1);
outputs = y(2:end,1);

%
TrainFcn = 'trainlm';


HiddenLayerSize = 1;

Net = fitnet(HiddenLayerSize, TrainFcn);

Net.inputs{1}.processFcns = { 'removeconstantrows' };

Net.outputs{2}.processFcns = { 'removeconstantrows' };


Net.divideFcn = 'dividerand';  % Divide data randomly
Net.divideMode = 'time';  % Divide up every sample
Net.divideParam.trainRatio = 70 / 100;
Net.divideParam.valRatio = 15 / 100;
Net.divideParam.testRatio = 15 / 100;
Net.trainParam.epochs = 2000; 

Net.performFcn = 'mse';  % Mean Squared Error

% Train the Network
[Net, tr] = train(Net, inputs', outputs','useParallel','no','useGPU','no' );

genFunction(Net, 'ANNFcn_Sine', 'MatrixOnly', 'yes');

ITS_Sine = ANNFcn_Sine(inputs')';

plot(ITS_Sine)

MSE=sum((outputs-ITS_Sine).^2)/length(y);

Cortexys set up:

 %%%%%%%%%%%%%%%%%%%%%%%%% Layer Setup %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 layers.af{1} = [];
 layers.sz{1} = [input_size 1 1];
 layers.typ{1} = defs.TYPES.INPUT;

 layers.af{end+1} = tanh_af(defs, []);
 layers.sz{end+1} = [6 1 1];
 layers.typ{end+1} = defs.TYPES.LSTM;

layers.af{end+1} = LinU(defs, defs.COSTS.SQUARED_ERROR);
layers.sz{end+1} = [output_size 1 1];
layers.typ{end+1} = defs.TYPES.FULLY_CONNECTED;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Pls give more details on your RNN-LSTM and your plain NN, e.g number of hidden units, number of layers, etc for apple-to-apple comparison. $\endgroup$
    – horaceT
    Commented Jul 13, 2016 at 20:46
  • $\begingroup$ I've added code to show how the neural network was generated. It is producing good results even with one hidden node! $\endgroup$
    – Baz
    Commented Jul 13, 2016 at 20:57
  • $\begingroup$ For the lstm I'm using cortexsys I'm getting good results with as little as six nodes. I've added the layer set up used in cortexsys. $\endgroup$
    – Baz
    Commented Jul 13, 2016 at 20:59
  • 1
    $\begingroup$ Your problem is too simple. It's a 1-dim series, and you didn't even add noise. Even with noise, such a fit could be accomplished in OLS with a couple of non-linear terms. Try something more complex,e.g. a high-dim time series with shifting trends, etc. $\endgroup$
    – horaceT
    Commented Jul 13, 2016 at 21:08
  • $\begingroup$ Do you have a specific example? Perhaps a process with a high dimensional ARMA as the base generator might allow the lstm to display its full potential? Or if you can think of some other process that would show off the lstm to full effect? $\endgroup$
    – Baz
    Commented Jul 13, 2016 at 21:18

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.