# Time series analysis with neural networks

I'm new to neural networks and machine learning and I was wondering how you use time series data to set the weights of a regular FNN, and how you use the ending weights to forecast the time series. In essence how can you transform the time series data into weights and back again for the output.

• Have you tried fitting a simple model like $v_k = \alpha v_{k-1} + \epsilon$ to your data? I'm all for neural networks, but I'd try a simple model like that if just to give yourself a baseline for comparison before looking at NNs. Simple $AR(k)$ models are good for things with oscillations too. Once you've hit the limits of this sort of "white box" model, then I'd think about NNs. – Patrick Caldon May 31 '12 at 4:04

A feed-forward neural network (typically multi-layer) is a type of supervised learner that will adjust the network weights on its input and internal nodes, in an iterative manner, in order to minimize errors between predicted and actual target variables. It commonly uses stochastic gradient descent (sometimes called error back propagation) over many iterations in order to find a local minimum of the error response and optimize the network weights accordingly.

The basic idea behind stochastic gradient descent is to start by randomizing the weights, then adjust them by iterating through several passes and updating the weights in a direction that moves the total error between target and predicted errors towards the local minimum error of the gradient surface. In practice, a tradeoff is found between optimizing a training set against a validation set, in order to reduce the problem of over-fitting.

Lastly, the input (time series or otherwise) often needs to be transformed in order to create a stationary series that is also bounded (amplitude wise) between the input range of the NN layer transfer function(s)(typically, 0 to 1 or -1 to 1).

Once the weights have been trained, the model can be stored and used to process additional new time series data, much like a typical linear regression based model.

An example illustration of using a NN to predict finanacial time series data, using Weka, is posted here: http://intelligenttradingtech.blogspot.com/2010/01/systems.html

A good text comparing financial AR based models against NN models is, "Applied Quantitative Methods for Trading and Investment," Christian Dunis et.al.

First, I believe that when you say "weight" you actual mean "input"/ "output". This is because you asked how to transform the time series to weights and how to transform output weights into a prediction. Neural network terminology uses "weight" to mean something else (pat's answer uses the term "weight" correctly).

This is what people usually suggest: If your time series looks like

X_1, X_2, ..., X_n, ...


then you do the following:

Step 1: Decide how many observations you want to use to make a prediction.

Step 2: Decide how many steps forward you want to predict.

Both these choices are fixed for the NN.

For this example let's say you want to use the last 5 readings to make 2 predictions.

Then you will

Step 3. Create a neural network with 5 input nodes and 2 output nodes.

Step 4. Create your training set with each element consisting of 5 sequential readings as input and the next two readings as output.

Here are the first TWO elements of the training set:

Input = X_1, X_2, X_3, X_4, X_5
Output = X_6, X_7

Input = X_2, X_3, X_4, X_5, X_6
Output = X_7, X_8

etc.


Now for some gyan.

If your data is noisy e.g. stock ticks, then my feeling is that this will be hard to train. I know I have had bad luck trying to train neural networks on noisy data.

So here is another strategy:

First model your time series using the ARIMA framework. This views a time series as

Polynomial base +
Cyclic component +
Bounded randomness


(Take a look at the Weka example in pat's answer from this point of view.)

Now my feeling (and I am still experimenting) is that the random component(s) are interfering with the training of the NN. So I want to avoid trying to predict them directly.

Picture your series data coming in. On every reading you feed it into your ARIMA black box, which figures out the underlying model and then spits out the ARIMA model parameters. So at time 0 you have a set of parameters, and then at time 1 you have an updated set of parameters, etc.

Note: The ARIMA black box is slow.

Question 1: Would it be possible for a neural network to learn how these parameters change? My feeling is that they will change slowly, so this may be doable.

Question 2: Could you train a different neural network to discern any patterns in the ARIMA error? I.e. If ARIMA predicts 5.4 and the actual next reading is 5.5, could you train a neural network to figure out that 0.1?

If you have access to a MATLAB installation, try the neural network toolbox first: http://www.mathworks.com/help/toolbox/nnet/gs/f9-56659.html (have a look at the screenshots)

It is very, very good for what you're trying to accomplish, and with great documentation. It is a great starting point.

There are several ways in which you can 'train' a neural network. Personally, I prefer the genetic algorithm approach - each individual represents a set of weights, with the fitness function being the performance of the neural network.

The performance of the neural network in terms of time series analysis could be the mean squared error of the predictions against the targets. One common method in time series prediction with neural networks is to use the % change from specific intervals over a 'lookback' period.

You may find this useful - http://ijcai.org/Past%20Proceedings/IJCAI-89-VOL1/PDF/122.pdf

• Your answer doesn't address the OPs main issue, the use of time series data. – Michael Chernick Aug 29 '12 at 16:28