Support vector regression for multivariate time series prediction

Question

Has anyone attempted time series prediction using support vector regression?

I understand support vector machines and partially understand support vector regression, but I don't understand how they can be used to model time series, especially multivariate time series.

I've tried to read a few papers, but they are too high level. Can anyone explain in lay terms how they would work, especially in relation to multivariate time series?

EDIT:To elaborate a bit, let me try to explain with a stock price example.

Say we have stock prices for N days. Then, for each day we could construct a feature vector, which, in a simple case, could be be the previous day's price and the current day's price. The response for each feature vector would be the next day's price. Thus, given yesterday's price and today's price the objective would be to predict the next days price. What I don't understand is, say we have six months training data, how would you give greater emphasis to the more recent feature vectors?

Answer 1

In the context of support vector regression, the fact that your data is a time series is mainly relevant from a methodological standpoint -- for example, you can't do a k-fold cross validation, and you need to take precautions when running backtests/simulations.

Basically, support vector regression is a discriminative regression technique much like any other discriminative regression technique. You give it a set of input vectors and associated responses, and it fits a model to try and predict the response given a new input vector. Kernel SVR, on the other hand, applies one of many transformations to your data set prior to the learning step. This allows it to pick up nonlinear trends in the data set, unlike e.g. linear regression. A good kernel to start with would probably be the Gaussian RBF -- it will have a hyperparameter you can tune, so try out a couple values. And then when you get a feeling for what's going on you can try out other kernels.

With a time series, an import step is determining what your "feature vector" ${\bf x}$ will be; each $x_i$ is called a "feature" and can be calculated from present or past data, and each $y_i$, the response, will be the future change over some time period of whatever you're trying to predict. Take a stock for example. You have prices over time. Maybe your features are a.) the 200MA-30MA spread and b.) 20-day volatility, so you calculate each ${\bf x_t}$ at each point in time, along with $y_t$, the (say) following week's return on that stock. Thus, your SVR learns how to predict the following week's return based on the present MA spread and 20-day vol. (This strategy won't work, so don't get too excited ;)).

If the papers you read were too difficult, you probably don't want to try to implement an SVM yourself, as it can be complicated. IIRC there is a "kernlab" package for R that has a Kernel SVM implementation with a number of kernels included, so that would provide a quick way to get up and running.

Answer 2

My personal answer to the question as asked is "yes". You may view it as a pro or a con that there are an infinite number of choices of features to describe the past.Try to pick features that correspond to how you might concisely describe to someone what the market has just done [eg "the price is at 1.4" tells you nothing if it is not related to some other number]. As for the target of the SVM, the simplest are the difference in prices and the ratio of prices for two consecutive days. As these correspond directly to the fate of a hypothetical trade, they seem good choices.

I have to pedantically disagree with the first statement by Jason: you can do k-fold cross-validation in situations like that described by raconteur and it is useful (with a proviso I will explain). The reason it is statistically valid is that the instances of the target in this case have no intrinsic relationship: they are disjoint differences or ratios. If you choose instead to use data at higher resolution than the scale of the target, there would be reason for concern that correlated instances might appear in the training set and validation set, which would compromise the cross-validation (by contrast, when applying the SVM you will have no instances available whose targets overlap the one you are interested in).

The thing that does reduce the effectiveness of cross-validation is if the behavior of the market is changing over time. There are two possible ways to deal with this. The first is to incorporate time as a feature (I've not found this very useful, perhaps because the values of this feature in the future are all new). A well-motivated alternative is to use walk-forward validation (which means testing your methodology on a sliding window of time, and testing it on the period just after this window. If behaviour is changing over time, the saying attributed to Niels Bohr "Prediction is very difficult, especially about the future" is especially appropriate. There is some evidence in the literature that the behaviour of financial markets does change over time, generally becoming more efficient, which typically means that successful trading systems deteriorate in performance over time.

Good luck!

Answer 3

There's an example up on Quantum Financier for using an SVM to forecast financial series. It could easily be converted from a classification system (Long/Short) to a regression system.