What is the frequency in my series and does it matter for my forecasting models (SARIMA)?

Question

I am working on multiple time series consisting of various sensor outputs. I am having data for more than 3 months duration. I am supposed to find p,d,q orders and coefficient values using the latest 200 values for the first time; and later update the model coefficients (while keeping the model order fixed as found from the 200 values) using the latest 100 values, at every 500 data points duration.

I use auto.arima (R 3.3.2) for finding the orders and coefficients for the first time (from 200 latest values) and then use Arima to update coefficients after every 500 values (from the latest 100 values), keeping the orders the same as found by auto.arima.

I am supposed to forecast one-step ahead and find residual at every new coming streaming sensor value.

The time series frequency is either 10 readings/minute, 1 reading/minute or 1 reading/10 minutes. As I have to consider the latest 200 values to decide orders, my time series data is either of these three cases:

1. separated by seconds and spread over 20 minutes;
2. separated by minutes and spread over 3 hrs and 20 minutes;
3. separated by 10 minutes and spread over 33 hrs and 33 minutes.

Also, for some instances I get

• 5, 6, 7, 9, 11 readings in place of 10 readings/minutes,
• 2 readings in place of 1 reading/minute and
• 2 readings/10 minutes or 1 reading/11 minutes in place of 1 readings/10 minutes.

Provided this, my questions are:

1. What should be the frequency parameter for the ts function for the given 3 types of time series?
2. Is it feasible to predict seasonality from 200 past values, or, we can avoid considering seasonality, as model parameters are getting updated at every 500 points?
3. What will be the "season' for these three cases? Can I consider season as an hour?

Answer 1

1. The frequency argument should be determined by subject-matter knowledge, and you have not told us much about that. The key point to consider is the period/frequency of the signal in your data. For example, quarterly macroeconomic data have frequency=4. Meanwhile, daily sales data may have frequencies 365.25 (if the day of the year matters - it would for ice-cream sales but maybe not for sales of tooth paste), 7 (if the day of the week matters - it would for almost any product), etc. The latter example shows that you may have multiple frequencies, and that cannot be handled directly with arima (but could be handled with tbats or by adding seasonal dummies of Fourier terms via xreg in arima).
   Edit: Given that By visual observation of data, no significant trend or seasonality component is found and if we trust the visual observation (you could do some tests or visualize the frequency decomposition as an alternative), then you can set frequency=1. This argument is used for considering seasonal models such as SARIMA, but if there is no seasonality, then frequency does not really matter. In absence of seasonality, it would neither matter that output frequency don't remain same for some of the sensors.
2. What matters is how many seasons are covered rather than how many observations you have. For example, if frequency=4, 200 observations should be more than enough to capture the seasonal patterns. But if the frequency=100, then 200 observations would be too few, i.e. you would heavily overfit trying to identify a seasonal pattern from just two full seasons worth of observations.
3. It depends on the application and is related to the answer to your first question.

Answer 2

The problem with memory modelling is that in many if not most circumstances there are unspecified deterministic structures/effects in play. For example one would never IMHO use a frequency of 365.25 because what happened 365.25 days ago is never really what is important. Many systems ( such as the # of people going to a clinic ) are driven by hourly , daily habits and monthly habits as we normally don't duplicate what we do from year to year on a weekly basis , although it can happen . Note however that a specific day within a month can be very important and we have even found cases where the week in the month was important and of course month-end effects. This is why subject matter knowledge can be important in setting the software switches to look for specific structure.

Additionally events like holidays and special sales days are important. I have dealt with many series ( from economic to physical) that can be characterized by a hybrid of deterministic effects and memory effects. These models are normally more heavily dependent on deterministic effects and usually only slightly dependent on memory (usually short-term memory) . Most times there are level shifts and time trends that need to be identified and conditioned for besides the pulses .

I suggest that you use a scheme where you have time series within a set of classes. For example all the readings for hour 1 can be used to predict hour 1 while incorporating a daily total variable as an X variable to help account for a lower frequency. In this example you would have 25 models. As was suggested you need to determine what the minor frequency is ( 24 -hourly in my example) and what the major frequency is ( 7 in my example) .