I'm barely a student in his internship, so I don't have a lot of experience, but I am facing an issue with a project I'm having at my work. I'm a bit in the dark because I find it hard to understand what is a satisfactory analysis given the situation, so please bear with me a few more lines. If the question is not relevant for this community, excuse my mistake and let me post it in on StackOverflow or take it down.
I need to provide "decent" forecasts of monthly sales in FMCG company. Expectations are not high, since it is known that some series have missing data or even multiple consecutive months without sales(that is a 0 value data point). Best series, so far, have around 28 months of sales. I use an extremely unorthodox method, but one cannot just perform 5000 forecasts manually. The way I approach this is as follows:
- I created 2 functions that choose best model fits for SARIMA and ets() wrapper to find best models per series for each family according to AIC(I use MAPE)
- I chose the model fit that scores best from each of the model family (SARIMA/ETS) and perform out of sample forecasting
- Then, I compare the out of sample accuracy measures(MAPE/RMSE) between the 2 and choose the best scoring one with its parameters or orders
- Ultimately, I forecast the desired horizon with the specifications(sarima order or ets arguments) that i obtained above
- When new data comes in(variously scattered during a month), my code automatically takes in consideration the newest data points and thus, updates the models. )
My method does not take in consideration assumptions, normality of data, homogeneity of residuals. I know this is quite counter-intuitive, but I'm considering to create a score/rank or so to filter out unacceptable forecasts.
My data is imported from SQL Server to RStudio. Here, the code splits each data set into lists of train and test sets that will be used for choosing a model. I use only 3 points for out of sample in the first few months as some series have 24+3=27 data points and I wanted a better capture of seasonality(relevant?). After choosing the best model, I perform the forecast which will be ultimately pushed into SQL Server to be used for reporting/visualizations.
[![rough depiction of the flows]]
- Since the "project" will yield forecasts for the next 12 periods and updates every month(model, accuracy, predicted values). Do you think 26-27 are plenty data points for the moment? Or do I simply have to accept a lower prediction accuracy or generalizability?
- Is there be a method to filter out models that are extremely dangerous(hideously non-normal series distribution, a sort of ranking based on the various out of sample accuracy measures and AICc/BIC?
- What could there be improved?