Modelling count data with time-series structure and predictors

Question

I am doing an analysis of sales-data over a period of time (i.e. over a few years). Those sales-data are also dependent on some predictive variables (i.e. holiday, weekend, weather,...). The daily count has a range of 0 to 10.000 and some zero-values. The problem I am confronted with the choice of a suitable predictive model, that includes the time-structure and the presence of zero values.

I have tried at first a poisson model. The resulting problem was a large overdispersion. To handle the overdispersion I did a Quasi-Poisson- and Negative-Binomial-Model. But here I have the problem of the time-series structure and the poor-predication of zeros (in general the models have a poor prediction-power). For that reason I considered a zero-inflated Poisson-Model (to handle the zeros). Nevertheless the model choice is very poor (so the predictions).

I hope that someone has a idea of a suitable model choice for my modelling problem (and how to handle the ts-structure). The actual results of the models I did, don´t suffice the data.

Answer 1

If you have only some zeros and frequent high counts, then you may be able to use models for continuous data, even though your data are count. For instance, consider ARIMAX or regression with ARIMA errors (the two are not the same).

Best approach for count prediction in time-series? is related. However, I essentially write that I have not seen any count data models that include autoregression or similar "TS" structure. You may be able to "roll your own" negative binomial regression model with a mean that depends both on predictors and autoregressively on previous values. Negative Binomial Regression by Hilbe might be useful, but he doesn't consider time series models.

Shameless self-promotion: I wrote a little article (Kolassa, 2016, IJF) on forecasting sales count data with regressors which may be useful. However, for all the reasons above, I didn't model any autoregression. Then again, it's probably not necessary for my application, which is supermarkets and drug stores that serve a couple thousand households - even if each household has an autoregressive demand for toilet paper, I would expect the autoregression to disappear in the aggregate data generating process. Your problem may be similar, or different.

Finally, if you are unsatisfied with the quality of your predictions, How to know that your machine learning problem is hopeless? may be helpful.

Answer 2

I think a natural choice of model is a state space model, ideally one that can handle both evolving latent states and external predictors

If you are familiar with python I would highly recommend having a look into this toolbox (NB I have no affiliation with anyone involved with it!):

https://github.com/mattjj/pylds

It allows you to fit time-series models for count data and has extensive support for over/underdispersed/zero inflated count data and allows you to include external predictors that can both affect the how latent processes evolve, as well as predictors for each timepoint. I feel like this matches pretty perfectly what you need.

Their inference techniques are very sophisticated and the implementations are blazing fast. I would definitely give it a try!

Answer 3

Daily sales data is often rich with opportunities to form a useful model. One example is here Simple method of forecasting number of guests given current and historical data and R Time Series Forecasting: Questions regarding my output AND here https://autobox.com/capable.pdf slide 49+ . Modelling count data ( both large and small !) is done all the time ... see the classic airline data set for an example of count data for large numbers http://www.autobox.com/pdfs/vegas_ibf_09a.pdf

In general the issue is to form a SARiMAX model as discussed here How to predict the next number in a series while having additional series of data that might affect it? . Often times the unspecified but important X's have to be found empirically via schemes like those presented here http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html

In my opinion to avoid arima/memory structure OR the detection of latent structure such as pulses , level shifts , local tine trends or seasonal pulses is to tacitly ignore structures that are often/usually necessary in order to compose a sufficient model..