4
$\begingroup$

I have time series data, collected hourly. I want to find if there are any patterns that repeat monthly. Most articles about this suggest using ACF or Fourier transforms, but that wouldn't work well for say a pattern that repeats every 15th of the month, or say every third weekend of the month. Are there any techniques that would help with this, without writing specific code to search for every possible pattern that repeats monthly?

$\endgroup$
6
  • $\begingroup$ I would suggest changing your title to "Finding monthly repeating patterns using daily data" $\endgroup$
    – IrishStat
    Sep 29, 2015 at 12:15
  • $\begingroup$ bearded has hourly data, so that phrase would be more informative. $\endgroup$
    – Nick Cox
    Sep 29, 2015 at 13:15
  • $\begingroup$ He might have data taken on a minute by minute basis or even on a second by second basis BUT if he wishes to identify specific daily patterns then in my opinion he need to get out of the weeds so to speak and bucket his data at a higher level of aggregation but that's just my opinion as one who has modelled thousands of time series problems like this. $\endgroup$
    – IrishStat
    Sep 29, 2015 at 13:32
  • $\begingroup$ The very small issue is just which is the better word; there is no challenge to your immense authority or experience and no tacit argument about analysis strategy. OP says hourly data. $\endgroup$
    – Nick Cox
    Sep 29, 2015 at 13:58
  • $\begingroup$ Thanks all for the suggestions below. I'll think my way through them, as well as read some of the linked material before I add anything to this thread. I apologize for not adding any details about the type of data - I'm legally bound not to do so. However, I can add one detail: I might be able to get meaningful daily aggregates - I am already removing any weekly patterns and can thus operate on residuals. $\endgroup$
    – bearded
    Sep 29, 2015 at 17:02

3 Answers 3

2
$\begingroup$

Whatever your software is, you should be able to calculate

  1. day of the month 1 to 31

  2. day of month measured from the end of the month (so 28, 27, 26 February in non-leap years would be -1, -2, -3)

  3. day of the week (using whatever convention appeals about which day starts a week) 1 to 7.

Then plot any or all of those variables -- on the $x$ axis, conventionally -- and the raw data (or the residuals from some model that sweeps out other variations, at least roughly) -- on the $y$ axis, conventionally.

Having hourly data as well is naturally a complication. Without knowing more substantive details than you give (essentially none), there are many possibilities. In principle hourly data can clearly also be plotted against time of month or time of week. Whether daily cycles should be averaged out is a judgment call.

Similarly without substantive detail it is difficult to advise on the role of dates of importance in business, religious and national holidays, etc.

These techniques are simple but often effective and surprisingly often neglected. Even moderately experienced workers with time series often have just one graphical idea, to plot against calendar date or date-time as a single series.

See also literature on cycle plots (other names exist). A friendly way in is Naomi B. Robbins' Introduction to cycle plots. She has various examples of weekly cycles. Although focused on detecting seasonal variation, my paper Graphs for all seasons may also be suggested.

$\endgroup$
2
  • $\begingroup$ W.r.t the following: "Then plot any or all of those variables -- on the x axis, conventionally -- and the raw data (or the residuals from some model that sweeps out other variations, at least roughly)". If I understand this correctly: say I have 6 months of data. The X axis would be 1:31, and y axis would have 6 * 24 points - 6 months * 24 points per day. I can do that - but after plotting it, what should I look for in the graph? How will it help me detect say something that happens every 1st? $\endgroup$
    – bearded
    Sep 29, 2015 at 19:05
  • $\begingroup$ You're looking for peaks or troughs on the graph. If the first day of the month is associated with highs or lows, then that should be evident on the graph. As in the literature cited, you may need to do a bit of work e.g. summarize or smooth to see patterns. With hourly data nothing stops you constructing an axis that is day of month + hour/24, or whatever. $\endgroup$
    – Nick Cox
    Sep 29, 2015 at 19:45
1
$\begingroup$

It is possible and practical to detect and incorporate the factors that you describe into a useful daily model. See this post Multivariate Forecasting and this web article http://www.autobox.com/cms/index.php/afs-university/intro-to-forecasting/doc_download/53-capabilities-presentation starting at slide 45 shown here as an introductory piece .enter image description here . I have found models that require week-of-the-month indicators and of course particular days-of-the-month indicators along with "Friday before a Monday Holiday effect" and "Monday-after a Friday Holiday effect" and/or "long weekend effects" . Daily data can be rich in information and it can often be excavated/mined/found via good analytics while compensating for anomalous data and ARIMA structure.

I would add user-suggested variables to the mix such as "third weekend in the month" if I had a premonition in that regard otherwise "cause effects" like these could be difficult to find in a purely empirical way. Having said that effects like these can often be found hiding in the residuals from an "inadequate model" thus revealing themselves for subsequent resolution. This is called residual diagnostic checking or more adequately "when did my model not produce clean (information free) residuals" .

Statistical analysis can often be aided by domain knowledge or myths that are whispered about by the elders in the village.

$\endgroup$
0
$\begingroup$

This is truly difficult. The thing with Fourier transforms, ACFs and other generic methods is that they will work on evenly sampled data. Conversely, the 15th of the month, "third week-end of the month" are usually not separated by the same timespan.

In this case, you should write code, as dummy variables, per example, for patterns you suspect.

It is not that hard though. You have to write a function:

bool[] = f(date_time)

which returns a vector of attributes. The good thing doing this is that you can return as much information as you wish : is it a bank day ? is it a Sunday ? Is it week end #1 in the month/in the year ? Is it a lunch break in my country ?

Then, depending on you approach, interactions may be evaluated automatically (is it lunch time on a week end ?)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.