1
$\begingroup$

I'm new to the neural network field and I would like to understand how one can backtest a neural network trained with backpropagation methodology.

Particularly, I have a time series dataset and I trained a neural network by using the neuralnet package in R; initially, my idea was to divide the dataset up 2 sub-samples:

  1. the Training Set, constituted by the 70% of available data;
  2. the Test Set, constituted by the remaining 30% of data;

and use the training set to develop the model and the test one for backtesting purposes. Of course, I thought to take those ones chronologically neighbouring.

Browsing on the internet, I found that the cross-validation methodology is used in this field; although there are different views about the way implementing this methodology, most ones suggest to divide the dataset up 5 sub-samples and use 4/5 for training and the rest to test the net.

I wonder if such methodology can be applied reasonably to neural network models and if it has sense applied to time series data.

Can you suggest what backtesting methodology adapts better to times series data?

$\endgroup$
1
$\begingroup$

Neural Networks are just one type of learner, falling under the much broader class of statistical/machine learning models. Most machine learning literature and examples tend to focus more on data that are IID (independent and independently distributed), and often sample and partition the data into several sets comprised of independent observations. It is common to use 10 fold cross validation with this in mind.

However, for a time series (particularly, financial time series, back or forward testing), that might not be suitable, since it does not take time series dependence or stability into consideration. What is often used in practice is walk-forward validation$^1$, which segments the train/validation sets into contiguous time slices. That not only allows us to validate our models over several slices, but in addition, it allows us to observe if the phenomena is stable over long periods of time, and also allows us to moniter and modify the model in a dynamic and online fashion, if warrented.

A simple way to modify the neural network would be to either, manually partition the segments into contiguous blocks, or see if the learner package you are using has that pre-processing capability. I've often found the bigger challenge is to formulate the model features and outcomes in some manner that a neural network or any other type of machine learner can be of beneficial use (not an easy feat at all).

1. From Wikipedia:

Walk forward optimization is a method used in finance for determining the best parameters to use in a trading strategy. The trading strategy is optimized with in sample data for a time window in a data series. The remainder of the data are reserved for out of sample testing. A small portion of the reserved data following the in sample data is tested with the results recorded. The in sample time window is shifted forward by the period covered by the out of sample test, and the process repeated. At the end, all of the recorded results are used to assess the trading strategy.1

I haven't seen it covered very much in academic or statistical/MLliterature (see R. Hyndman's recent work), but it is covered a lot more in modern trading literature and books. e.g.

The Evaluation and Optimization of trading systems. R. Pardo

Quantitative trading systems, H. Bandy

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.