0
$\begingroup$

I'm using SVM and (neural network) for a time series prediction data-set in MATLAB R2016a with 800 samples. Currently I'm using 10-fold cross validation and grid search to find best SVM parameters. I'm using 90 samples (after this 800 samples) as out-of-sample to check performance of final model using best SVM (and neural network) parameters and training my model on whole first 800 samples.

The test accuracy of final model (10-fold cross validation) is about 98% (sensitivity and specificity of about 98%) but when I check designed model on last 90 out-of-sample data (which trained using whole first 800 samples) I have a poor accuracy (about 55~59% total accuracy, sensitivity and specificity). This is daily forecasting of a financial market. Why I have this behavior? I checked normal k-fold cross validation and sliding window validation (discussed below in comments). I had mentioned behavior (poor out-of-sample accuracies) in two methods.

$\endgroup$
  • 1
    $\begingroup$ accuracy is not a proper scoring rule, so it will select the wrong model with high probability. Use a proper scoring rule like log-likelihood or brier score. $\endgroup$ – Sycorax May 16 '16 at 13:49
  • $\begingroup$ @C11H17N2O2SNa. I got same results with bier score in SVM model. why? any suggestion? $\endgroup$ – user2991243 May 16 '16 at 14:58
1
$\begingroup$

You need to use time-series cross-validation to tune your models, rather than random cross-validation.

e.g. (borrowing from Rob Hyndman's blog):

library(fpp) # To load the data set a10
plot(a10, ylab="$ million", xlab="Year", main="Antidiabetic drug sales")
plot(log(a10), ylab="", xlab="Year", main="Log Antidiabetic drug sales")

k <- 60 # minimum data length for fitting a model
n <- length(a10)
mae1 <- mae2 <- mae3 <- matrix(NA,n-k,12)
st <- tsp(a10)[1]+(k-2)/12

for(i in 1:(n-k))
{
  xshort <- window(a10, end=st + i/12)
  xnext <- window(a10, start=st + (i+1)/12, end=st + (i+12)/12)
  fit1 <- tslm(xshort ~ trend + season, lambda=0)
  fcast1 <- forecast(fit1, h=12)
  fit2 <- Arima(xshort, order=c(3,0,1), seasonal=list(order=c(0,1,1), period=12), 
      include.drift=TRUE, lambda=0, method="ML")
  fcast2 <- forecast(fit2, h=12)
  fit3 <- ets(xshort,model="MMM",damped=TRUE)
  fcast3 <- forecast(fit3, h=12)
  mae1[i,1:length(xnext)] <- abs(fcast1[['mean']]-xnext)
  mae2[i,1:length(xnext)] <- abs(fcast2[['mean']]-xnext)
  mae3[i,1:length(xnext)] <- abs(fcast3[['mean']]-xnext)
}

plot(1:12, colMeans(mae1,na.rm=TRUE), type="l", col=2, xlab="horizon", ylab="MAE",
     ylim=c(0.65,1.05))
lines(1:12, colMeans(mae2,na.rm=TRUE), type="l",col=3)
lines(1:12, colMeans(mae3,na.rm=TRUE), type="l",col=4)
legend("topleft",legend=c("LM","ARIMA","ETS"),col=2:4,lty=1)
$\endgroup$
  • $\begingroup$ Thank you for answer. I used both normal and your proposed cross validation techniques but I have same behavior. $\endgroup$ – user2991243 May 16 '16 at 13:56
  • $\begingroup$ I used this technique: fold 1 : training [1:n-50], test [n-50:n-40] ~~ fold 2 : training [1+10:n-40], test [n-40:n-30] and so on. This is just an example with imaginary numbers. Is this the same method? If not, please describe more about your technique. I'm not familiar with R. $\endgroup$ – user2991243 May 16 '16 at 19:53
  • $\begingroup$ I checked porposed method in : stats.stackexchange.com/questions/14099/… but I have same problem. $\endgroup$ – user2991243 May 16 '16 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.