0
$\begingroup$

I would have expected this to be covered in detail by the LIBSVM tutorial but after hours of wasting time googling for answers I've had to throw in the towel. What I am trying to do is rather trivial really. Right now I have a financial data set in CSV format:

time,indicator1,indicator2,indicator3,roc
20140508190000,0,-2,-1,-0.00361310835712725
20140508200000,1,-2,0,0.0216786501427154
20140508210000,-1,-2,1,-0.0252917584998426
20140508220000,0,-2,0,-0.0108397167220604
20140508230000,-1,-2,0,-0.0505725535527163
201405090,0,-2,0,0.0144560896277541
2014050910000,0,-2,0,-0.00361363061468619
2014050920000,1,-2,0,0.0542122953485871
2014050930000,-1,-2,1,-0.0975645009756423
2014050940000,0,-2,1,-0.0144550448106373
2014050950000,-1,-2,1,-0.198670712324811
...

So we have the date/time column and then four attributes per row. Later on I am going to want to predict the last column (roc). This data is obviously not normalized and I intend to use phraug to convert my data into a LIBSVM friendly format. But not so fast...

Here's the problem. I have not been able to find any examples that explain how to format multi-row input data. In other words, since I plan to use SVR I somehow need to feed my SVM a sliding window of data sets. For instance, let's say I want to give the engine five rows in order to have it predict the output value of the sixth:

20140508190000,0,-2,-1,-0.00361310835712725
20140508200000,1,-2,0,0.0216786501427154
20140508210000,-1,-2,1,-0.0252917584998426
20140508220000,0,-2,0,-0.0108397167220604
20140508230000,-1,-2,0,-0.0505725535527163

The ideal here would be the roc value of the next row: 0.0144560896277541

Then we slide the window down by one row:

20140508200000,1,-2,0,0.0216786501427154
20140508210000,-1,-2,1,-0.0252917584998426
20140508220000,0,-2,0,-0.0108397167220604
20140508230000,-1,-2,0,-0.0505725535527163
201405090,0,-2,0,0.0144560896277541

And the ideal here is: -0.00361363061468619

Etc. - in essence I need to feed LIBSVM sets of data, not just atomic rows of data. Either I can just keep my original format (normalized) and configure LIBSVM's training/prediction routine to do that somehow, OR I need to convert my five rows into one row, and I have no clue how to do that. Actually I really hope that this is done via some kernel hyperparam and not via the CSV. Otherwise it would require a different CSV input file for differing ranges of data sets. And that would be very complicated and error prone.

I hope all this made sense. Can someone PLEASE point me in the right direction? Any input would be much appreciated.

UPDATE:

I think I'm starting to see the light! Just to clarify - I have no problem with HOW to format my data once I fully understand the needed format. Let's assume I use regression and a window of 3 days of data with three input values per day. Per my understanding this involves creating a multi-dimensional vector, thus this would be the proper input matrix for a 3x3 vector:

response_variable 1:day1-1 2:day1-2 3:day1-3 4:day2-1 5:day2-2 6:day2-3 7:day3-1 8:day3-2 9:day3-3

For SVC my response variable would most likely be an int representing a category, e.g. up/down (-1/1) and for SVR something like a normalized ROC (e.g. 0.014).

Can someone please confirm this? Thanks in advance! :-)

$\endgroup$
  • $\begingroup$ Using sliding window to classify them is generally not very good idea. $\endgroup$ – Vladislavs Dovgalecs May 13 '15 at 6:28
1
$\begingroup$

Yes you need to turn your 5 rows into 1 row, by changing the csv file. I don't know how you are calling libsvm, but scikitlearn/Matlab/... etc have APIs so you can pass the regenerated matrices in (rather than creating separate csv files for varying number of lags). The lag generation might be done in pandas? Alternatively you might want to look at Rapidminer or other gui tool for doing the whole transformation and looping through different lags.

$\endgroup$
  • $\begingroup$ Thanks for the quick response. Right now I'm calling LIBSVM via the command line for training but eventually I'll be using the C# lib in my trading app. For now I just need to figure out the data format. Now, you said that I need to produce a long single column. How does LIBSVM know how to separate them all? Or is it agnostic to that? If I produce a single row of 5x5 params that'll get me to 25 then - does that include all date columns/params? And I assume I need the ideal at the beginning of the row, correct? $\endgroup$ – Michael Aug 25 '14 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.