Correct procedures to detect and correct outliers for aggregated/SKU time series

Question

Background

I am currently working with sets of product sales time series at SKU-level for a FMCG company. Data are available in a weekly format for multiple years and sales data for hundreds of products are available at any single point of time. Products are very similar in nature. They are seasonal with moving seasonal components (easter etc.) and generally have a product life of less than 2-3 years. However, not all products share the same sales pattern, some even have intermittent demand. There are almost no labels (or additional information) apart from the original time series.

Purpose

The mission is to develop a methodology to remove promotional effect and real outliers of historic data at SKU-level to create baseline sales history for other purposes.

Concerns

Without much industrial experience I am struggling to figure out a good way to tackle - nor did I manage to find literatures that specifically address this issue. I am not even quite sure about where it should start - should the first steps be looking at SKU sales or the aggregated sales?

Products are launched consistently throughout the year and their life time are so short that there is almost always not enough historical data for a single product. In the light of this and due to lack of information, the current methodology aims to identify, and correct the outliers based on a single aggregated time series and corrects time series at SKU-level based on results from that. The first part is done in an interesting fashion that I am not sure whether the results are actually statistically valid and I believe it is unlikely to be better than simply using regression to estimate the parameters. However, how are we going to adjust the data at SKU-level based on the corrected data at aggregated-level?

I would like to kindly enquire what kind of procedures do businesses / would you take and reasonings behind it. Furthermore, are there any literatures that might be useful for this? Please advice me if there is anything else that you need to know or would like to be clarified.

Thank you.

Answer 1

I don't recommend pooling as there is no information suggesting a common model across the dis-aggregates. I would attempt to form a separate model for each dis-aggregate that incorporated seasonal effects and memory. Seasonal effects can be either based upon memory (ARIMA) or dummy seasonal factors. There may be level shifts or time trends in the data and/or some unusual data points due to anomalies or price/promotion effects. Weekly data can be effected by what day of the week holidays fall and the holidays themselves. Non-causal i.e. purely ARIMA approaches often fail to deal with weekly data because what we did last year at week i is different than what is done this year at week i. Each series (dis-aggregate) will have it's own story/pattern and one has to approach the problem with a many-faceted solution tool that can distinguish data generating functions and not simply fit a pre-specified model. I have been busy these many years (nearly 50) in trying to develop ever-better procedures. The way one approaches the identification of each model incorporating memory,dummies and user-specified causals can be (is) critical. I would suggest that you actually investigate both free procedures and commercial procedures to effectively solve this.