NARX architecture of neural networks

Question

I’ve a question related to the NARX architecture of neural nets:

What is the usage of the tapped delay line in this architecture?

My problem is, that I can’t figure out the difference of this compared to a basic multilayer perceptron. Let’s say I have a set of data $S$ which represents a time series. I always want to predict the $k_{th}$ values of this series by training the network on subsets of $S$ from $j$ to $k-1$. So, the amount of tapped delay lines of the network is equal to the size of values of the subsets of $S$. On predictions of the $k_{th}$ value, I backpropage the error between the expected and the predicted value to adjust the weights. For the next prediction, I start with the expected kth value to be the last in the subset. So what I actually do is to use a sliding window over my training data where the expected, the target value is always feed back into the net with when the sliding window moves.

Can’t I just use a MLP for this? Is it not just a matter of the training procedure?

Answer 1

First of all the use of the tap delay line in this architecture is to express an input vector that is composed of the time-series data of one recorded variable.

So if you have N number of data in a set S, consider t to be an index of a given data-point at time t. Now if your defined delay is d, meaning how many time-steps in the past you are incorpoparting in the tap delay (in other words how far back in the past you want to look) then your Tap Delay Line(TDL) is:

TDL=[s(t), s(t-1), s(t-2)..., s(t-d)]

Now off course d < N otherwise you would not be able to build a TDL as there will be not enough data in the past. If now you want to train your network the target of this input vector would be the time-step:

target = s(t+1)

If you are predicting then the prediction is pushed into the TDL like so:

TDL=[s(t+1), s(t), s(t-1)..., s(t-(d-1))]

Example

Imagine you have time-series data set S from a sensor.

S = [0.3, 0.5, 0.7, 0.9, 0.11, 0.13] and delay d = 2. Then if we want to create a TDL for s(t) = 0.11 (t=4, t being the index of the array) the result is:

TDL = [0.11, 0.9, 0.7] and the target is target=[0.13].

Mathematicaly the NARX architecture is based on Takens's theorem.

In theory any regression model can implement a NARX architecture but MLPs are the most common. So to answer your question you can use an MLP to implement a NARX.

This paper is very useful if you want more detail on NARX.

Hope this helps!