I am currently working with sets of product sales time series at SKU-level for a FMCG company. Data are available in a weekly format for multiple years and sales data for hundreds of products are available at any single point of time. Products are very similar in nature. They are seasonal with moving seasonal components (easter etc.) and generally have a product life of less than 2-3 years. However, not all products share the same sales pattern, some even have intermittent demand. There are almost no labels (or additional information) apart from the original time series.


The mission is to develop a methodology to remove promotional effect and real outliers of historic data at SKU-level to create baseline sales history for other purposes.


Without much industrial experience I am struggling to figure out a good way to tackle - nor did I manage to find literatures that specifically address this issue. I am not even quite sure about where it should start - should the first steps be looking at SKU sales or the aggregated sales?

Products are launched consistently throughout the year and their life time are so short that there is almost always not enough historical data for a single product. In the light of this and due to lack of information, the current methodology aims to identify, and correct the outliers based on a single aggregated time series and corrects time series at SKU-level based on results from that. The first part is done in an interesting fashion that I am not sure whether the results are actually statistically valid and I believe it is unlikely to be better than simply using regression to estimate the parameters. However, how are we going to adjust the data at SKU-level based on the corrected data at aggregated-level?

I would like to kindly enquire what kind of procedures do businesses / would you take and reasonings behind it. Furthermore, are there any literatures that might be useful for this? Please advice me if there is anything else that you need to know or would like to be clarified.

Thank you.

  • 1
    $\begingroup$ Thank you for going to the effort of crafting a readable, informative question. Welcome to our site! $\endgroup$ – whuber Jun 25 '15 at 14:47
  • $\begingroup$ I can not imagine how adjustments/price promotion/outliers developed on an aggregate could be efficiently applied to the dis-aggregates. My experience tells me that adjustments can be made at both levels prior to reconciliation. Reconciliation is normally done in either a top-down or a bottoms-up approach using forecasting accuracy to determine/suggest which gambit us best. $\endgroup$ – IrishStat Jun 25 '15 at 14:58
  • $\begingroup$ Very well developed question. What you are after is called top-down or bottom up forecasting. There is a nice introductory textbook on this topic by Hyndman et al. - Forecasting principle and practice. This covers this topic. With regards to outlier correction and promotion response, there are few specialist in this website, I hope they see your question and respond. I have added few tags that might get their attention. $\endgroup$ – forecaster Jun 25 '15 at 14:58
  • $\begingroup$ I must admit that I feel a bit overwhelmed to see three regulars commenting in my every first post - It's a great pleasure to meet you all and @whuber , thank you so much for your warm welcome. $\endgroup$ – WeakLearner Jun 26 '15 at 0:47
  • $\begingroup$ @IrishStat For the first part - If starting from (a huge set of) dis-aggregates of unique products in different stages of life cycles, what would be a good approach to identify the outliers? Would you suggest having a look at techniques such as pooling? (I've also started investigating automated packages in R). Thank you for sharing your great thoughts! $\endgroup$ – WeakLearner Jun 26 '15 at 0:47

I don't recommend pooling as there is no information suggesting a common model across the dis-aggregates. I would attempt to form a separate model for each dis-aggregate that incorporated seasonal effects and memory. Seasonal effects can be either based upon memory (ARIMA) or dummy seasonal factors. There may be level shifts or time trends in the data and/or some unusual data points due to anomalies or price/promotion effects. Weekly data can be effected by what day of the week holidays fall and the holidays themselves. Non-causal i.e. purely ARIMA approaches often fail to deal with weekly data because what we did last year at week i is different than what is done this year at week i. Each series (dis-aggregate) will have it's own story/pattern and one has to approach the problem with a many-faceted solution tool that can distinguish data generating functions and not simply fit a pre-specified model. I have been busy these many years (nearly 50) in trying to develop ever-better procedures. The way one approaches the identification of each model incorporating memory,dummies and user-specified causals can be (is) critical. I would suggest that you actually investigate both free procedures and commercial procedures to effectively solve this.

  • $\begingroup$ Thank you for your elaborative response. To examine and support issues you mentioned I think I will (learn and) use DTW to perform clustering for set of time series that the company thinks behave in a similar fashion. $\endgroup$ – WeakLearner Jun 26 '15 at 15:47
  • $\begingroup$ That could be useful in segmenting the time series into similar groupings prior to performing the magic that I detailed above. $\endgroup$ – IrishStat Jun 26 '15 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.