# Forecasting daily time series sales revenue with many zero entries

I have been trying to forecast the sales revenue of different product groups (the displayed sales revenue is aggregated over all products for each day e.g. smartphones with different prices as one group) but haven't found the right approach yet. I am pretty much a novice in time series analysis / forecasting and use Python. I tried using (S)ARIMA but didn't get meaningful results. I think the main issue is that my data is too sparse and I only have about a year and a half of data points. On top of that the time series has high fluctuations. Is my assumption correct that ARIMA has issues with data structured like mine? (Since it uses the mean of series?)

The top image is one of the "nicer" timeseries and the bottom an "average" one. I tried just resampling to weekly data and then I think the forecast fit better but I lose some information with averaging over the week. For the bottom example weekly data was still not enough to use ARIMA. I would rather use daily data if possible. In the end I would like to add exogenous features like day of the week, weather, promotion etc

Given the count like nature of my data what options do I have? I would really appreciate some pointers. My research lead to "Crostons method". Does it fit to my problem? Would I need to change my $into sales numbers #? Is there a Python package for croston's or do I have to use R? Any help is much appreciated. pastebin data EDIT To clarify what the different time series represent: Both are different product groups (I have about 10-20). And the forecasting I am more interested in (since most of my data looks like that) is the bottom one. I did some more preparation for the bottom time series. from statsmodels.tsa.stattools import adfuller series = dataframe["Sales Revenue2"] X = series.values result = adfuller(X) (-1.3334675205365911, 0.6137252422831784, 15, 267, {'1%': -3.4550813975770827, '10%': -2.5725712007462582, '5%': -2.8724265892710914}, 3623.772858172633)  So the time series is not stationary. Now using decompose:  from statsmodels.tsa.seasonal import seasonal_decompose result = seasonal_decompose(dataframe["Sales Revenue2"], model='additive') fig = result.plot() plt.show()  Honestly I have not figured out the decomposition 100%. Does the picture show me have a weekly seasonality since the patterns repeats every 7 days? Next looking at the ACF and PACF What does the pattern tell me? And that? I also tried to difference the time series and got def difference(dataset, interval=1): diff = list() for i in range(interval, len(dataset)): value = dataset[i] - dataset[i - interval] diff.append(value) return diff  PACF looks very funky. (negative values increasing from 41 to 48 outside conf interval to -1, then jumping to positive 3 at 49, slowly decreasing afterwards) ACF was inside conf interval apart from a negative value at order 1 My next step was to try to use auto.arima (in Python) I tried m=7, for the weekly seasonality? and different start values. What I don't get is why those were the orders which fit best? How does that compare to my first ACF/PACF - would I need to rather dif t=7 to see the real ACF, PACF of that order? I also tried to fit without seasonal trend setting m=1. What irritated me the most is that order (1,1,1) got very close to the best model without seasonality (12,1,1) (how does that order come together?) The AICs where 3743 compared to 3738 I would really appreciate if someone could guide me a bit through my results. Can I use ARIMA to forecast the data I have? The new results do not look too bad. I was mostly getting results similar to the last example just around the mean. But I was using stepwise before and now I explicitly tried higher orders. What if I have even less non zero values. What would my approach be to convert a time series to count data? Thanks again! (Since I don't have enough reputation I had to cut some graphs. The first time series is now gone since it doesn't really focus on my topic - I just wanted to show a comparison. I also had to cut the differenced ACF and PACF • Your data is very spikey, which will cause problems (your time series model would have to jump up and down a lot). You may have to use higher order MA processes to smooth out the series. Oct 16, 2018 at 11:33 • I tried using auto.arima first. It never even considers the higher orders, usually stops around 0 or 1. Resulting in very unspikey predictions. Should I try it by hand? Is the AIC here not usable? Should I rather look into ACF and PACF? Oct 16, 2018 at 12:24 • auto.arima works good, but domain knowledge works better. You should always check ACF and PACF, never trust an automatic model blindly. Also, why does your data look like this? Is there some underlying process which explains the spikes? Some event(s) that are related (which could cause the spikes)? Oct 16, 2018 at 12:49 • The data looks like this because the product price is mostly around 100 dollars up to$500. The sale quantity is very low. Every time a few pricey items get sold there is a spike, There is a very small correlation to promotions/sales percentage but I would say it is mostly the nature of the shop Oct 16, 2018 at 12:53
• The reason I am asking is because these spikes may just be a product of chance, in that there is no time component (trend) in your data. It may be that it all comes down to the shop and the way they do their business, or these are just random fluctuations. Since you have few observations you could try binning the results into larger intervals and see if a pattern emerges, for ex. binning by month, then the interval september-october looks interesting. Otherwise you could convert this time series into a count time series and use an appropriate analysis for such data. Oct 16, 2018 at 13:25

Would I need to change my $into sales numbers #? Ideally yes. Since this is not a unique product, how do I know if 600\$ is one 600\$unit or 2 300$ units?

Is there a Python package for crostons or do I have to use R?

Right now there are not any good R packages for Croston's and similar intermittent methods. R offers better options.

Is my assumption correct that ARIMA has issues with data structured like mine?

Depends. ARIMA might work with the data in the top graph, but it won't work with the data in the bottom graph (or anything sparser than that).

Croston's and it's newer versions (TSB, etc...) are a better option. But you need to keep in mind that such methods don't produce a normal forecast the way ARIMA or ETS does. They forecast a rate of sale (or velocity) which can then be used then to figure out average sales over a long period of time.

You should also look at count predicting methods like Negative Binomial and Poisson distributions.

• So if I were to use a count approach I would need to bin my data too?0-100,100-200 etc?Since I am also interested in the total revenue,As you said 12*$50 is different to1*$600.Or would that make my data too sparse?Negative Binomial and Poisson were terms I picked up during my "research". Do you have a good starting point for application to time series? In the end I would like to forecast the next week, using external factors like weather, promotions etc for the different product groups. What time period are you talking about when using the forecast rate?Can it forecast like 2 sales next week? Oct 17, 2018 at 7:57
• I wouldn't recommend binning the data, but actually using the unit sales time series instead of the revenu time series (I assume you have access to it). Then you can calculate the expected revenu off of your forecasts. Typically you forecast units, not revenu. Oct 17, 2018 at 17:23
• For using Poisson and Neg Bin, I have mostly seen it in academic papers. That's where I would start. For example: researchgate.net/profile/Keith_Ord/publication/… Oct 17, 2018 at 19:03
• See also here: rpubs.com/ashvenkat/flu_timeseries Oct 17, 2018 at 21:15

I encountered the same problem recently with a time serie similiar to yours, kept trying different models but the one that worked the best for this kind of series is using XGboost with hyperparameter optimization and some lagged features.