I am working on multiple time series consisting of various sensor outputs. I am having data for more than 3 months duration. I am supposed to find p,d,q orders and coefficient values using the latest 200 values for the first time; and later update the model coefficients (while keeping the model order fixed as found from the 200 values) using the latest 100 values, at every 500 data points duration.

I use auto.arima (R 3.3.2) for finding the orders and coefficients for the first time (from 200 latest values) and then use Arima to update coefficients after every 500 values (from the latest 100 values), keeping the orders the same as found by auto.arima.

I am supposed to forecast one-step ahead and find residual at every new coming streaming sensor value.

The time series frequency is either 10 readings/minute, 1 reading/minute or 1 reading/10 minutes. As I have to consider the latest 200 values to decide orders, my time series data is either of these three cases:

  1. separated by seconds and spread over 20 minutes;
  2. separated by minutes and spread over 3 hrs and 20 minutes;
  3. separated by 10 minutes and spread over 33 hrs and 33 minutes.

Also, for some instances I get

  • 5, 6, 7, 9, 11 readings in place of 10 readings/minutes,
  • 2 readings in place of 1 reading/minute and
  • 2 readings/10 minutes or 1 reading/11 minutes in place of 1 readings/10 minutes.

Provided this, my questions are:

  1. What should be the frequency parameter for the ts function for the given 3 types of time series?
  2. Is it feasible to predict seasonality from 200 past values, or, we can avoid considering seasonality, as model parameters are getting updated at every 500 points?
  3. What will be the "season' for these three cases? Can I consider season as an hour?

2 Answers 2

  1. The frequency argument should be determined by subject-matter knowledge, and you have not told us much about that. The key point to consider is the period/frequency of the signal in your data. For example, quarterly macroeconomic data have frequency=4. Meanwhile, daily sales data may have frequencies 365.25 (if the day of the year matters - it would for ice-cream sales but maybe not for sales of tooth paste), 7 (if the day of the week matters - it would for almost any product), etc. The latter example shows that you may have multiple frequencies, and that cannot be handled directly with arima (but could be handled with tbats or by adding seasonal dummies of Fourier terms via xreg in arima).
    Edit: Given that By visual observation of data, no significant trend or seasonality component is found and if we trust the visual observation (you could do some tests or visualize the frequency decomposition as an alternative), then you can set frequency=1. This argument is used for considering seasonal models such as SARIMA, but if there is no seasonality, then frequency does not really matter. In absence of seasonality, it would neither matter that output frequency don't remain same for some of the sensors.
  2. What matters is how many seasons are covered rather than how many observations you have. For example, if frequency=4, 200 observations should be more than enough to capture the seasonal patterns. But if the frequency=100, then 200 observations would be too few, i.e. you would heavily overfit trying to identify a seasonal pattern from just two full seasons worth of observations.
  3. It depends on the application and is related to the answer to your first question.
  • $\begingroup$ Thanks Richard. I am dealing with 171 sensor outputs, and they have different output frequencies. Some of the output plots contains noise which might be due to abnormal working of sensors. By visual observation of data, no significant trend or seasonality component is found. Also, output frequency don't remain same for some of the sensors. If we update the model frequently, can we avoid considering seasonality all together? $\endgroup$
    – na ja
    Commented Nov 15, 2016 at 15:45
  • $\begingroup$ @nj, I have updated my answer to address your comments. $\endgroup$ Commented Nov 15, 2016 at 17:48
  • $\begingroup$ is there a way I can share my data with you? email id? $\endgroup$
    – na ja
    Commented Nov 16, 2016 at 12:59
  • $\begingroup$ @nj, I am not ready to do data analysis for you (unlike IrishStat, I guess). But I am willing to answer concrete questions and this way not only help you with your problem but also increase the value of Cross Validated for future users. Let me know if you have more questions or need clarification for the answers to the existing ones. $\endgroup$ Commented Nov 16, 2016 at 13:20
  • $\begingroup$ that's appreciated! I don't want you to do my work. I wanted to share the my R script plus data, so that, what I was trying to explain by seasonality could be formulated in a better manner. I think, i need to post code and data to stack-overflow. $\endgroup$
    – na ja
    Commented Nov 16, 2016 at 16:06

The problem with memory modelling is that in many if not most circumstances there are unspecified deterministic structures/effects in play. For example one would never IMHO use a frequency of 365.25 because what happened 365.25 days ago is never really what is important. Many systems ( such as the # of people going to a clinic ) are driven by hourly , daily habits and monthly habits as we normally don't duplicate what we do from year to year on a weekly basis , although it can happen . Note however that a specific day within a month can be very important and we have even found cases where the week in the month was important and of course month-end effects. This is why subject matter knowledge can be important in setting the software switches to look for specific structure.

Additionally events like holidays and special sales days are important. I have dealt with many series ( from economic to physical) that can be characterized by a hybrid of deterministic effects and memory effects. These models are normally more heavily dependent on deterministic effects and usually only slightly dependent on memory (usually short-term memory) . Most times there are level shifts and time trends that need to be identified and conditioned for besides the pulses .

I suggest that you use a scheme where you have time series within a set of classes. For example all the readings for hour 1 can be used to predict hour 1 while incorporating a daily total variable as an X variable to help account for a lower frequency. In this example you would have 25 models. As was suggested you need to determine what the minor frequency is ( 24 -hourly in my example) and what the major frequency is ( 7 in my example) .

  • $\begingroup$ Thank you Dave. As we are dealing with sensor output, it stays within fix range.But, the data is noisy. No trend or seasonality components are detected while eyeballing the plots of data. $\endgroup$
    – na ja
    Commented Nov 15, 2016 at 15:40
  • $\begingroup$ Then your problem is "easier" , If you want to post your data I will try and tale a "look" at it. $\endgroup$
    – IrishStat
    Commented Nov 15, 2016 at 15:44
  • $\begingroup$ Regarding the yearly frequency, it is really a matter of application. I have worked with data where the time of year was an important determining factor, so 365.25 was just what was needed. Note that I did not suggest to look at what happened 365.25 days ago (as in autoregressive modelling) but rather treat the same time of the year in the same way over multiple years (as in Fourier terms or seasonal dummies). $\endgroup$ Commented Nov 15, 2016 at 16:00
  • $\begingroup$ time of year is often very important and is often better handled by combinations of deterministic factors like hour-of-the-day; day-of-the-week,week-of-the-month;month-of-the-year,periods before and after events;long-weekends;end-of-month;particular days-of-the-month. Comparing what you do today with a year ago doesn't yield much information as we don't follow that pattern BUT we do respond to the items I have heretofore mentioned. Fourier procedures simply don't cut the mustard as they over-fit and render residuals that are far from random which is why residuals need to be examined .. $\endgroup$
    – IrishStat
    Commented Nov 15, 2016 at 16:13
  • $\begingroup$ @Dave, data file is rather big. can I mail you the same? $\endgroup$
    – na ja
    Commented Nov 15, 2016 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.