4
$\begingroup$

I'm applying standard 0-1 scaling of features before SVM classification for financial data but the results are worse. This is the results before scaling

    NORMAL DATA AVERAGE RESULTS
      Profit           PF         avMC         avPP         avRC        totTP        totFP         PF>1     algosnum           SS          SSl
  4389060.90         6.85        -0.00        60.69         0.50        16086        10973            5            8            1            5

and this is after scaling

NORMAL DATA AVERAGE RESULTS
      Profit           PF         avMC         avPP         avRC        totTP        totFP         PF>1     algosnum           SS          SSl
  2256204.80      2044.51        -0.07        52.53         0.46        14577        12220            4            8            1            5

Scaling is performed in 0-1 range, test data is scaled according to scaling factor of train data. From the above results you can see that precision went down (avPP) from 60.69 to 52.53, average Mathew Correlation Index from 0 to -0.07 number of true positives went down from 16086 to 14577 and number of false positives grown from 10973 to 12220. The presented result is an outcome of 80 classifications on different financial instruments data for 80 data sets 20000x200 so i think result is quite significant.

So my question is: In such situation how I should proceed? Shall I stick to scaling? Or maybe I should generate different data set to check if this behavior is consistent? What sort of analysis of my features I can make? My data set is a mix of binary and continuous features in different scales.

Improve this question
$\endgroup$
8
  • $\begingroup$ Before scaling, what's the smallest value you have? What about after? Are you sure you're not losing numerical precision after scaling? $\endgroup$
    – Alex R.
    Commented Sep 16, 2015 at 18:44
  • $\begingroup$ the smallest 0.001, the biggest 200, the variables are double type so i think its OK $\endgroup$
    – Krzysztof Fajst
    Commented Sep 16, 2015 at 18:59
  • $\begingroup$ what happens if you run it on a small random subset of your data? Something like 10 points for example. $\endgroup$
    – Alex R.
    Commented Sep 16, 2015 at 21:08
  • $\begingroup$ What you mean ? There is 80 test sets each has 1440x200 points and 80 train sets 20kx200 each. On 1st 10 points of test sets ?? Anyway im creating new data sets now from different time period so i will know if this behavior is consistent at least $\endgroup$
    – Krzysztof Fajst
    Commented Sep 16, 2015 at 21:59
  • 1
    $\begingroup$ what you mean 'update SVM' ?? I scale the train data than test data with the same scaling factor $\endgroup$
    – Krzysztof Fajst
    Commented Sep 19, 2015 at 16:59

1 Answer 1

Reset to default
6
$\begingroup$

Keep in mind why people typically scale features prior to estimating an SVM. The notion is that the data are on different scales, and this happenstance of how things were measured might not be desirable -- for example, measuring some length quantity in meters versus kilometers. Obviously one will have a much larger range even though both represent the same physical quantity.

However, there's no reason to believe that the new scaling is any better. While it's true that the rescaled features will all vary in comparable units, it's also possible that the original scaling happened to encode the data such that some important features had more prominence in the model.

You don't mention what kernel function you're using, but I think it's illustrative to consider the example of two different versions of the Gauissian RBF kernel: $K_1(x,x^\prime)=\exp(-\gamma||x-x^\prime||^2_2).$ This is an isotropic kernel, meaning that the same scaling ($\gamma$) is applied in all directions. A more general kernel function might have the form $K_2(x,x^\prime)=\exp\big(-(x-x^\prime)\Gamma(x-x^\prime)\big);$ it is anisotropic as $\Gamma$ is a diagonal PSD matrix, with each element applying a different scaling to each direction. The advantage of this kernel function is that it will vary more strongly in some directions than others.

Coming back to your question, it's possible to imagine that your data have, for whatever reason, some features that are more important than others, and that this coincides with the scale on which they are measured. Placing them on the new scale where they all appear on similar scales and are all treated as equally important means that unimportant or noise features cloud the signal.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm using SPegasos solver from WEKA package ttic.uchicago.edu/~nati/Publications/PegasosMPB.pdf $\endgroup$
    – Krzysztof Fajst
    Commented Sep 17, 2015 at 9:53
  • $\begingroup$ To be more specific how i got those results. Initially data is classified if buying or selling is profitable, than small ensemble is created from buy and sell signal for every instrument and based on this new trade signal is generated. Than this signal is evaluated as a new classification for every instrument and all trading days. Than results are averaged. So its not straight forward classification as ensemble and also some smoothing is involved. I'm double checking now but i think this behavior is consistent so it happens always regardless from which time period i will take a data. $\endgroup$
    – Krzysztof Fajst
    Commented Sep 17, 2015 at 10:05

Not the answer you're looking for? Browse other questions tagged or ask your own question.