Yes I face the age-old problem of sales forecasting for a large enough company. Mostly to determine our staffing needs for various functions. So I'm only really interested in aggregate data, not individual product SKU forecasts like most businesses.
Here's the challenge. Traditional time series methods like exponential smoothing, Holt Winters, triple ETS with dampening, do not seem good enough. They're easy enough to understand and implement. However they are entirely backwards looking, of course. The assumption is the system will continue to move, and change, as it has moved, and changed, and that's that.
If you happen to see a brick wall, or rocket fuel, on the horizon, the time series model does not care, or account for this, of course.
So I'm thinking about adding some kind of regression equation, or at least maybe blend a regression equation with the time series (don't feel like anything overly complicated like an ARIMA with regressors which is likely not the right model anyway).
For instance, we're using salesforce and have indicators of quantities x % close rates, that may, in part, help forecast future growth.
Here's my issue with coming up with a model that takes into account historical data, and past seasonality (very strong time-of-year influences) with this future data (sales team picture of leads, lead stage, probability, size).
I can't just add the two together. The time series model >already accounts< for some latent growth in the model. By which I mean, each year, old business may expand in size, and new business may be landed. The model already accounts for this.
It's hard for me to "backwards separate" previous years' 'Salesforce' data -- because that data was never collected/ doesn't exist.
So how do I combine these models? Is it a factor of separating components (level a, trend b, seasonality g) --- and only using the Salesforce data as a modifier to the trend component?
But also, the Salesforce data is only new business, it doesn't account for old business growth --- as a result, should I separate those two items out in the forecast as well?
All this seems like an EXTREMELY common sales forecast procedure that many sizes of companies may have to use, yet everywhere I look online, the forecasting advice is either : a painfully simple model that doesn't even begin to touch on seasonality or any complexity.
Or the opposite: a very statistically rigorous method that either ignores future judgment (time series) or discounts time patterns (regression) ... or overfits the data.
Any help in where to begin to tackle this problem?
To be clear, I have 6 years worth of very granular data (every product, client, category sold) --- but only need an aggregate forecast.
Time series fails because our growth has been slowing year after year to a halt (0%) ... the best-fit model extrapolates this quite well ... but it can never 'pre-empt' systemic changes or future events. It's inadequate. How do I combine regressors/ future judgement into this? Is it less formal than I think?