# Support vector regression for multivariate time series prediction

Has anyone attempted time series prediction using support vector regression?

I understand support vector machines and partially understand support vector regression, but I don't understand how they can be used to model time series, especially multivariate time series.

I've tried to read a few papers, but they are too high level. Can anyone explain in lay terms how they would work, especially in relation to multivariate time series?

EDIT:To elaborate a bit, let me try to explain with a stock price example.

Say we have stock prices for N days. Then, for each day we could construct a feature vector, which, in a simple case, could be be the previous day's price and the current day's price. The response for each feature vector would be the next day's price. Thus, given yesterday's price and today's price the objective would be to predict the next days price. What I don't understand is, say we have six months training data, how would you give greater emphasis to the more recent feature vectors?

-
 Can you post the papers you are referring to? – Quant Guy Aug 18 '11 at 21:58 Predicting Time Series with a Local Support Vector Regression Machine, Rodrigo Fernandez , svms.org/regression/Fern.pdf Support Vector Regression for Financial Time Series Forecasting, Wei Hao and Songnian Yu, springerlink.com/index/946802600u4552m8.pdf – raconteur Aug 19 '11 at 9:15

In the context of support vector regression, the fact that your data is a time series is mainly relevant from a methodological standpoint -- for example, you can't do a k-fold cross validation, and you need to take precautions when running backtests/simulations.

Basically, support vector regression is a discriminative regression technique much like any other discriminative regression technique. You give it a set of input vectors and associated responses, and it fits a model to try and predict the response given a new input vector. Kernel SVR, on the other hand, applies one of many transformations to your data set prior to the learning step. This allows it to pick up nonlinear trends in the data set, unlike e.g. linear regression. A good kernel to start with would probably be the Gaussian RBF -- it will have a hyperparameter you can tune, so try out a couple values. And then when you get a feeling for what's going on you can try out other kernels.

With a time series, an import step is determining what your "feature vector" ${\bf x}$ will be; each $x_i$ is called a "feature" and can be calculated from present or past data, and each $y_i$, the response, will be the future change over some time period of whatever you're trying to predict. Take a stock for example. You have prices over time. Maybe your features are a.) the 200MA-30MA spread and b.) 20-day volatility, so you calculate each ${\bf x_t}$ at each point in time, along with $y_t$, the (say) following week's return on that stock. Thus, your SVR learns how to predict the following week's return based on the present MA spread and 20-day vol. (This strategy won't work, so don't get too excited ;)).

If the papers you read were too difficult, you probably don't want to try to implement an SVM yourself, as it can be complicated. IIRC there is a "kernlab" package for R that has a Kernel SVM implementation with a number of kernels included, so that would provide a quick way to get up and running.

-
+1 for kernlab recommendation. – Wayne Aug 18 '11 at 20:25
@Jason, thanks for the response. I'm confused as to how to make older data less important during training, i.e. how to give important to the last say 100 examples, rather than the last 10,000 examples. Also, I have some experience working with SVMs in Python, so I am looking to use that. – raconteur Aug 19 '11 at 9:20
You can't really use all the past data as your feature vector; you have to transform it somehow into a meaningful feature. For example, consider averaging the past 20 values of the time series, or the current value of the series divided by the moving average. These are features that indicate in one simple number an import characteristic of the data set. Come up with (say) 10 or so features that you think will be as predictive as possible. So to make older data less important, you'd make it less important when defining your features (i.e. use shorter moving averages). – Jason Aug 19 '11 at 14:35
@Jason, let me try to explain with a stock price example. Say we have stock prices for N days. Then, for each day we could construct a feature vector, which, in a simple case, could be be the previous day's price and the current day's price. The response for each feature vector would be the next day's price. Thus, given yesterday's price and today's price the objective would be to predict the next days price. What I don't understand is, say we have six months training data, how would you give greater emphasis to the more recent feature vectors? – raconteur Aug 19 '11 at 15:26
Oh I see what you mean now. The short answer is, you can't, at least not without reengineering the SVM to understand the notion of recency. An SVM is defined in terms of x-y pairs, not ordered x-y pairs, and so the objective function it minimizes is a function of said pairs. OTOH, you could run several separate SVMs, one with the past month, one with the past year, and one with the past two years -- and then do a weighted average of the responses you get back based on your personal preference for recent vs. old data (e.g. 0.7 * past month + 0.2 * past year + 0.1 * past 2 years). – Jason Aug 20 '11 at 0:28

There's an example up on Quantum Financier for using an SVM to forecast financial series. It could easily be converted from a classification system (Long/Short) to a regression system.

-
 DEAR GOD PLEASE nobody use that strategy! That's how beginners get smoked by Wall St :( The oldest trick in the book is retrofitting a statistical model using some basic technical analytic like RSI so the newbs think they've struck gold. – Jason Aug 19 '11 at 14:37 @Jason from the original post "I also want to make clear that I do not think that this is a good system." However, it is a good example of how to fit a support vector machine to a time series. – Zach Aug 19 '11 at 15:11