What is the best way to strip weekly and seasonal noise from a time series data set?
Any recommendations on different approaches and there relative benefits?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
What you might actually be looking for is some way to remove the cyclical component in the data from the long term trend.
Any smoothing technique will be able to remove noise and the cyclical component in the data. A simple moving average or exponential smoothing technique is sometimes very useful. The difference between the predicted smoothed value and the actual value is the cyclical component in the data plus noise.
You can use a more complicated techniques such as a Hodrick–Prescott filter (usually applied to business cycles). I image you could also fit some function with a aggressive regularization component to encourage smoothness. The prediction by the smooth model fitted will be the trend component and the difference the seasonal component.
Alternatively, simple technique such as seasonality indexes might be used.
A seasonal index (quarterly sales data) is calculated in the following table.
Calculate what the hypothetical constant demand was in each quarter $i.e 403/4$. Calculate in what ratio the actual demand in quarter was above or below the constant demand $i.e 80/100.75$. Decompose data into the trend component $i.e 80/0.89$ and seasonal component $i.e 80-80/0.89$. If you have a monthly data consider converting the 12 data points per year to 4 data points with a weighted average. Basically taking the average of every three month to compute one data point. Weighing the start and end of season less than the middle of the season.
Alternatively, you can also build a simple OLS regression to separate the effect of each season from the long term trend. One independent variable is time, and other independent variables are dummy variables to indicate the season. Use $y=\alpha + \beta t+\beta_1 d_1+\beta_2 d_2 +\beta_2 d_3$ where $t$ is the time period starting from 1, and $d_1$, $d_2$ and $d_3$ is dummy variables for the quarter 2, 3, 4. Using quoter 1 as the base model, add the coefficient $\beta_2$ to all data-points in quoter 2, $\beta_3$ to all data-points in quarter 3, and $\beta_4$ to all data-points in quarter 4. All the data points will now effectively be in the same quarter (quarter 1). That is, seasonal component of different quarter has been removed.
The different between approaches comes down to effort and complexity. As in any modelling task, no penance exists. The most appropriate technique can only be chosen after examining the objective of what you are trying to do. A very comprehensive standard for seasonal adjustment of time series data is used in the EU. This document gives some very good guidelines. See ESS guidelines
I read this question differently than in @entropy's answer. I see this as a "double seasonality" request: i.e. the poster needs to filter out seasonal noise (e.g. a warm/cold cycle, modeled by week) and a weekly seasonal pattern (stores sell more on Saturday than on Tuesday).
There is a good discussion of this problem by Gould and Vahid-Araghi as chapter 14 in Rob J. Hyndman, A.B Koehler, J.K. Ord and R.D. Snyder's book 'Forecasting with Exponential Smoothing'. (2008). But the poster may be looking for a simpler approach.
Continuing my example above, convert the data to weeks, and estimate weekly seasonal factors (e.g. by time series decomposition). Use these weekly seasonal factors to deseasonalize the daily data. Then use this daily data to estimate daily seasonality, and remove that by deseasonalizing.