0
$\begingroup$

I'm barely a student in his internship, so I don't have a lot of experience, but I am facing an issue with a project I'm having at my work. I'm a bit in the dark because I find it hard to understand what is a satisfactory analysis given the situation, so please bear with me a few more lines. If the question is not relevant for this community, excuse my mistake and let me post it in on StackOverflow or take it down.

I need to provide "decent" forecasts of monthly sales in FMCG company. Expectations are not high, since it is known that some series have missing data or even multiple consecutive months without sales(that is a 0 value data point). Best series, so far, have around 28 months of sales. I use an extremely unorthodox method, but one cannot just perform 5000 forecasts manually. The way I approach this is as follows:

  1. I created 2 functions that choose best model fits for SARIMA and ets() wrapper to find best models per series for each family according to AIC(I use MAPE)
  2. I chose the model fit that scores best from each of the model family (SARIMA/ETS) and perform out of sample forecasting
  3. Then, I compare the out of sample accuracy measures(MAPE/RMSE) between the 2 and choose the best scoring one with its parameters or orders
  4. Ultimately, I forecast the desired horizon with the specifications(sarima order or ets arguments) that i obtained above
  5. When new data comes in(variously scattered during a month), my code automatically takes in consideration the newest data points and thus, updates the models. enter image description here)

My method does not take in consideration assumptions, normality of data, homogeneity of residuals. I know this is quite counter-intuitive, but I'm considering to create a score/rank or so to filter out unacceptable forecasts.

My data is imported from SQL Server to RStudio. Here, the code splits each data set into lists of train and test sets that will be used for choosing a model. I use only 3 points for out of sample in the first few months as some series have 24+3=27 data points and I wanted a better capture of seasonality(relevant?). After choosing the best model, I perform the forecast which will be ultimately pushed into SQL Server to be used for reporting/visualizations.

[![rough depiction of the flows][1]][1]

Questions, finally:

  1. Since the "project" will yield forecasts for the next 12 periods and updates every month(model, accuracy, predicted values). Do you think 26-27 are plenty data points for the moment? Or do I simply have to accept a lower prediction accuracy or generalizability?
  2. Is there be a method to filter out models that are extremely dangerous(hideously non-normal series distribution, a sort of ranking based on the various out of sample accuracy measures and AICc/BIC?
  3. What could there be improved?
Improve this question
$\endgroup$
1
  • $\begingroup$ Have you taken a look at the forecast package and its function of the same name? I think it was designed with exactly this kind of use case in mind. $\endgroup$ – ulfelder Feb 17 '20 at 15:26
1
$\begingroup$

Your approach sounds reasonable. With fewer than 28 historical data points, your options are limited, anyway.

Which gives you the answer for your first question: no, 28 monthly data points is not a lot. You have observed less than three years of data. Once you hold out a couple of data points for method selection or similar, you will be left with only two years, which is very little to fit a SARIMA model. (And yes, seasonality may well be relevant. Or not. It depends completely on what kind of consumer products you are forecasting. Toilet paper is non-seasonal. Sunscreen is seasonal.) Then again, many CPG products will not even be available for more than three years until they are superseded (in which case you may be able to stitch time series together).

So, you seem to be doing good things for a start. You may be able to do something more fancy by leveraging, e.g., product hierarchies and product similarities. However, that would require some deeper understanding of your data than we can likely provide here. You may profit from the excellent free online book Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman.

One simple "sanity check" would be to find series where your forecast is higher than the highest historical observation, or higher than the historical 90% quantile. This often signals trouble, except possibly for very new products that are ramping up. Ultimately, your sanity checks will depend on the kind of data you have.

One thing to keep in mind is that the MAPE may lead you towards biased forecasts, especially for slow moving products.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the insightful answer. The products are described by quite varied seasonality, promotions, discounts etc. I'm aware that it is impossible to account for all of the effects. My strategy is to forecast on aggregated brands(multiple products per brand), then calculate previous year's product share and split each brand accordingly to sum (100%) of the forecast. To combat the bias, I thought about putting in perspective other measures, like RMSE or so, and to create a formula. But even there, I do not want to add apples and pears in an equation. Would relying on AIC reduce bias? $\endgroup$ – Andrei Catana Feb 18 '20 at 7:13
  • 1
    $\begingroup$ AIC should reduce the bias, compared to MAPE. Given promotions, you may want to run regressions on these external factors and possibly model residuals with time series methods. Good luck! $\endgroup$ – Stephan Kolassa Feb 18 '20 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.