# Low performance of SVM (and neural network) in out-of-sample data with high test accuracy of 10-fold cross validation in a financial time series

I'm using SVM and (neural network) for a time series prediction data-set in MATLAB R2016a with 800 samples. Currently I'm using 10-fold cross validation and grid search to find best SVM parameters. I'm using 90 samples (after this 800 samples) as out-of-sample to check performance of final model using best SVM (and neural network) parameters and training my model on whole first 800 samples.

The test accuracy of final model (10-fold cross validation) is about 98% (sensitivity and specificity of about 98%) but when I check designed model on last 90 out-of-sample data (which trained using whole first 800 samples) I have a poor accuracy (about 55~59% total accuracy, sensitivity and specificity). This is daily forecasting of a financial market. Why I have this behavior? I checked normal k-fold cross validation and sliding window validation (discussed below in comments). I had mentioned behavior (poor out-of-sample accuracies) in two methods.

• accuracy is not a proper scoring rule, so it will select the wrong model with high probability. Use a proper scoring rule like log-likelihood or brier score. – Sycorax says Reinstate Monica May 16 '16 at 13:49
• @C11H17N2O2SNa. I got same results with bier score in SVM model. why? any suggestion? – user2991243 May 16 '16 at 14:58

You need to use time-series cross-validation to tune your models, rather than random cross-validation.

e.g. (borrowing from Rob Hyndman's blog):

library(fpp) # To load the data set a10
plot(a10, ylab="\$ million", xlab="Year", main="Antidiabetic drug sales")
plot(log(a10), ylab="", xlab="Year", main="Log Antidiabetic drug sales")

k <- 60 # minimum data length for fitting a model
n <- length(a10)
mae1 <- mae2 <- mae3 <- matrix(NA,n-k,12)
st <- tsp(a10)[1]+(k-2)/12

for(i in 1:(n-k))
{
xshort <- window(a10, end=st + i/12)
xnext <- window(a10, start=st + (i+1)/12, end=st + (i+12)/12)
fit1 <- tslm(xshort ~ trend + season, lambda=0)
fcast1 <- forecast(fit1, h=12)
fit2 <- Arima(xshort, order=c(3,0,1), seasonal=list(order=c(0,1,1), period=12),
include.drift=TRUE, lambda=0, method="ML")
fcast2 <- forecast(fit2, h=12)
fit3 <- ets(xshort,model="MMM",damped=TRUE)
fcast3 <- forecast(fit3, h=12)
mae1[i,1:length(xnext)] <- abs(fcast1[['mean']]-xnext)
mae2[i,1:length(xnext)] <- abs(fcast2[['mean']]-xnext)
mae3[i,1:length(xnext)] <- abs(fcast3[['mean']]-xnext)
}

plot(1:12, colMeans(mae1,na.rm=TRUE), type="l", col=2, xlab="horizon", ylab="MAE",
ylim=c(0.65,1.05))
lines(1:12, colMeans(mae2,na.rm=TRUE), type="l",col=3)
lines(1:12, colMeans(mae3,na.rm=TRUE), type="l",col=4)
legend("topleft",legend=c("LM","ARIMA","ETS"),col=2:4,lty=1)

• Thank you for answer. I used both normal and your proposed cross validation techniques but I have same behavior. – user2991243 May 16 '16 at 13:56
• I used this technique: fold 1 : training [1:n-50], test [n-50:n-40] ~~ fold 2 : training [1+10:n-40], test [n-40:n-30] and so on. This is just an example with imaginary numbers. Is this the same method? If not, please describe more about your technique. I'm not familiar with R. – user2991243 May 16 '16 at 19:53
• I checked porposed method in : stats.stackexchange.com/questions/14099/… but I have same problem. – user2991243 May 16 '16 at 21:27