0
$\begingroup$

I try to predict values for regression in LIBSVM. My data is in time series. I use gridregression.m file in LIBSVM to find optimal parameters c, g and p. Gridregression.m file use cross validation to find optimal parameters, but is it ok to use cross validation in time series?

When I use parameters from gridregression.m, sometimes the MSE is not better then the default values. ( cmd= '-s 3 -t 2' is sometimes better )

$\endgroup$
1
$\begingroup$

Standard cross validation (when you randomly split data into k blocks) is a wrong thing to do for time-series because in time-series you often have serial dependence, data are not iid.

For example, draw a random walk path, then cut-off one point in the middle, fit a regression on the rest and predict the value for the cut-off point - your prediction error will be small because cut-off point's value if very similar to neighbor values, it's not independent from them so cross-validation in this case will give you very optimistic estimate of prediction error.

Better thing to do to test a time-series regression is to run a moving 1-step-ahead prediction with fitting a regression on every step.

But even then, just calculating mean squared error is not good enough if your series aren't stationary - it's not very indicative. You may need to compare 1-step-ahead prediction error of your model and some simple reference model like "next step predicted value is the same as previous step realized value". This will indicate how good is you model out-of-sample compared to a reference.

$\endgroup$
0
$\begingroup$

Using cross-validation is fine, whatever the problem you are facing (regression, classification, time-series (which is just regression), ...). If your results after cross-validation are routinely worse than the default values, your cross-validation procedure is flawed.

One error that comes to mind is a poor choice of search grid. Does your search grid include the default values?

$\endgroup$
  • $\begingroup$ They are bigger than the defaults. I want to make them between [-15, 15] for each c, g and p, but it's so time consuming. So i use for c: 2^[-10,6], for g : 2^[5,-15] and for p 2^[15,0]. If the values is found in boundary, for i.e. c is 2^6, I search for c again between [4..8]. $\endgroup$ – user2602256 Jul 23 '13 at 20:04
  • $\begingroup$ @user2602256 c and gamma should never be negative! $\endgroup$ – Marc Claesen Jun 29 '14 at 11:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.