# Forecasting with large, high frequency dataset

I am doing my master's thesis and I must compare various forecasting techniques at different frequencies of datasets. I am using my universities dataset, the REDD dataset, UCI dataset and CER Ireland dataset for this purpose. The data I use is in seconds for a time span of a month and this gives > 3 million records.

I have been trying to understand how to make good use of all this data but couldn't exactly get to a solution. I have read Time series modeling with high-frequency data, but I don't understand and couldn't find resources how to apply it to my problem. I have tried reading several blogs and books to get an understanding of time series forecasting but most literature has examples with granularity only as low as hourly data. Some references I found were about high granularity data but only for a short period of time.

I have read in Prof. Rob Hyndman's blog that practically the ARIMA model can only calculate till 200 autoregressive points and if my understanding is correct then for data with a frequency in seconds, I could achieve daily trends only with $3600*24 = 86400$ previous values?

I am not sure how I should deal with this.

Here is how the data look (the y-axis is watts):

• Have you checked out Analysis of Financial Time-Series by Ruey Tsay? Chapter 5 of the book covers High-Frequency Data Analysis. Looks like you might find useful information there. I won't provide a link, but if you do a quick search you'll find it online. – Graeme Walsh Aug 6 '13 at 17:50
• Thank you Graeme Walsh. Yes I will check it out now but I was wondering if there is some guidance of how to do this with the "forecast" package. I have been reading the book "Hyndman and Athanasopoulos (2012), Forecasting: principles and practice, OTexts." but could not find an exact solution. I read that the tbats method support high frequency data but couldn't find much guidance on how to use them, improve them. – Rohit Tidke Aug 7 '13 at 9:24