ARIMA for daily data over 5 years - forecast package

Question

I have a question to auto.arima and seasonality. I have to analyze 39 single datasets which are prices of futures or equities. There are missing data which I replace with na.approx. Then I calculate the log return of the data with ln and lead. I split each data set into a in sample (2013/10/1 – 2017/09/30) and out of sample part (2017/10/01-2018/09/30).

I have to find an appropriate ARIMA Modell for each series which delivers good results in my foreacst. My calculated returns are stationary, tested it with ADF, KPSS and Philipps and Perron.

First of all, I used auto.arima function. With regard to ACF and PACF of the residuals some of the model suggestions where not good, since ACFs and PACFs showed of the residuals where significant or showed a clear pattern. For these datasets I created AIRMA manually. Now having all models, I used the forecast function in R to get results. My MASE and RMSE where pretty bad for my forecasts (with auto.arima and manual arima) and the results tend to zero. One example of one dataset where auto.arima provided good results:

d<-read.csv("MYFILE", header = TRUE, sep =";")
t<-as.Date(d$date,format="%d.%m.%Y")
require("tseries")
require("forecast")
 DJCT<-d$DJCT
 DJCTaprx<-na.approx(DJCT)
 DJCTretINSA<-ln(DJCTaprx[368:1828]/lead(DJCTaprx[368:1828], 1))[1:1460]
 DJCTretOUTSA<-ln(DJCTaprx[3:368]/lead(DJCTaprx[3:368], 1))
 DJCTret<-ln(DJCTaprx/lead(DJCTaprx, 1))
auto.arima(DJCTretINSA, stationary=TRUE)
Series: DJCTretINSA 
ARIMA(1,0,1) with zero mean 

Coefficients:
         ar1     ma1
      0.4807  -0.350
s.e.  0.1449   0.155

sigma^2 estimated as 7.947e-05:  log likelihood=4820.64
AIC=-9635.29   AICc=-9635.27   BIC=-9619.43
 arimaDJCTinsa<-auto.arima(DJCTretINSA, stationary=TRUE)

 plot(forecast(arimaDJCTinsa, h=365))

accuracy(forecast(arimaDJCTinsa, h=365), DJCTretOUTSA[1:365])
                        ME        RMSE         MAE  MPE MAPE      MASE
Training set -0.0001304150 0.008908426 0.006162879  NaN  Inf 0.8182688
Test set      0.0003074467 0.010278043 0.006968379 -Inf  Inf 0.9252182
                    ACF1
Training set 0.002305611
Test set              NA

Going through many blog posts of forecasting daily data and arima I wondered whether I had to add a seasonal component like fourier terms to improve my results. I used

tsDJCT<-ts(ln(DJCTaprx[368:1828]/lead(DJCTaprx[368:1828], 1))[1:1460], frequency=365)
bestfit <- list(aicc=Inf)
  for(i in 1:25)
  {
  fit <- auto.arima(tsDJCT, xreg=fourier(tsDJCT, K=i), seasonal=FALSE)
  if(fit$aicc < bestfit$aicc)
   bestfit <- fit
   else break;
   print(i)
   }
[1] 1

Or ets() function I always get 1 or no seasonality as result. Also findfrequency always showed (1) for all of my time series. Adding Fourier terms didn’t turn the AIC better.

 auto.arima(tsDJCT, xreg=fourier(tsDJCT, K=1), seasonal=FALSE)
Series: tsDJCT 
Regression with ARIMA(1,0,1) errors 

Coefficients:
         ar1      ma1  S1-365  C1-365
      0.4519  -0.3228  -7e-04  -3e-04
s.e.  0.1538   0.1633   4e-04   4e-04

sigma^2 estimated as 7.939e-05:  log likelihood=4822.33
AIC=-9634.67   AICc=-9634.63   BIC=-9608.24
fcDJCT<-forecast(auto.arima(tsDJCT, xreg=fourier(tsDJCT, K=1), seasonal=FALSE),xreg=fourier(tsDJCT, K=1), h=365)

accuracy(fcDJCT, DJCTretOUTSA[1:365])
                        ME        RMSE         MAE MPE MAPE      MASE
Training set -0.0001320885 0.008898123 0.006158973 Inf  Inf 0.8177503
Test set      0.0003072622 0.010246078 0.006997517 NaN  Inf 0.9290870
                    ACF1
Training set 0.001586422
Test set              NA

I tried to change the kind of time series and add the frequency with 365 which didn’t change my results.
Can anybody help me with the following questions?

• Was my way to find seasonality within my data right? So I should exclude it?

• Since I only got results for “no seasonality” I think I can exclude msts, right?

• Am I using auto.arima in a wrong way since my forecast results are that bad?

• How can I improve my results for arima for better forecasts if seasoanliyt or trend is not a factor?

Thank you!!! Find my example Dataset here https://www.dropbox.com/sh/p9q9r6mkpl1zidj/AAC7wsWUVNlu97JR-66Tn2Sfa?dl=0

Answer 1

Was my way to find seasonality within my data right? So I should exclude it?

Both using frequency to make your time series seasonal (and trusting auto.arima() to detect any actually relevant seasonality) and including Fourier terms are defensible. (They model different kinds of seasonality, though. Experiment with series simulated using both approaches if you are interested. See below on why I won't go into this further.)

Since I only got results for “no seasonality” I think I can exclude msts, right?

No. You might have seasonality on different frequencies, which your current approaches do not pick up, but methods for multiple-seasonalities might, if specified correctly. See below on why I won't go into this further.

Am I using auto.arima in a wrong way since my forecast results are that bad?

No, you are using auto.arima() correctly. See below (again).

How can I improve my results for arima for better forecasts if seasoanliyt or trend is not a factor?

Short answer: you can't.

Somewhat longer answer: How to know that your machine learning problem is hopeless?

Even longer answer: there is serious money to be had in forecasting financial returns. If it were as easy as setting up a good ARIMA model, many people would already be doing it, and any profits would already have been gone (the Efficient Markets Hypothesis in a very weak form). Also, anyone who already had such a magic bullet would almost certainly not tell you on CrossValidated.

There is a reason why ARIMA is not used seriously in financial forecasting.