Conditional restricted Boltzmann machines on a time series dataset

Question

Preamble of the problem

I am currently trying to apply Conditional Restricted Boltzmann Machines on a time series dataset problem, in particular, the dataset constitutes of 10 day stock market recordings, and in each day, 50 stock value recordings are logged that are 5 minute apart from each other. Therefore, to my presumption, each day can be considered as a sequence {x0,x1,...,x50} and each recording in a day is a sample x.

My presumptions - please correct me if I'm wrong

1. Each day of recordings is a sequence

2. The purpose of C-RBM is to map a sequence - a day of stock recordings - into a set of features that can be used by any regression algorithm as input

3. A different RBM is trained on each sequence

Question

How do we feed the sample recordings of a sequence to an RBM ? Do we concatenate all the features of a sequence into one large vector and train the RBM on that? Or do we use each sequence as a miniature dataset on which the RBM trains, but then the time element is not emphasized here?

The question basically boils down to how, from an abstract point of view, can we use C-RBM to extract hidden features from sequences?

Your suggestions will be highly appreciated.

Thank you

Answer 1

As I understood the CRBM its idea is to model high dimensional time series, for example from motion tracking. Thus each visible node of your (C)RBM at time t can be thought of as a datapoint from one of the time series (like node 1 is a datapoint from the hand time series, node 2 is a datapoint from the left foot time series and so on...). A sequence is than a certain amount of datapoints that you put together into a batch. Before you train, you should shuffle your batches to reduce autocorrelation you training set.

If you only have a single stock market time series than I think this wont work straight away, since there are for example no connections between the visible units within one RBM, so you wont be able to model any autoregressive dependencies. Furthermore the conditional independence, given the hidden units is not valid anymore. Additionally the connection between the RBM at time t and time t-m doesn't make sense anymore.

I would suggest you to read through the page of Graham Taylor again: http://www.cs.nyu.edu/~gwtaylor/publications/nips2006mhmublv/

A fix to your problem might be to use multiple stock time series (probably at a higher frequnecy so that you have enough observations) and than feed this into the code provided above. Something you should keep in mind is non-stationarity that often occurs especially with financial time series. To solve mean-value non-stationarity (trending data) you could simply take the returns instead of the raw prices.

Often the visible units are modeled as gaussian nodes with unit variance. I have my doubts that this is a good choice for financial time series. First of all think of ARCH/GARCH models, which try to "cure" this kind of non-stationarities of those time series. Secondly leptokurtosis is often exhibited, thus a gaussian distribution might not be the best choice (a better approximation could be a t-distribution). There is a paper by Welling, from 2005 which talks about on how to extend RBMs with nodes from the exponentional family of distributions.Maybe you want to have a look at it.