8
$\begingroup$

I have 3000 customers in my base and I want to forecast next 6 months revenue at a monthly level for each of these 3000 customers. Does that mean i have to build 3000 Arima models 1 for each customer?

I can build a automated process to try with different values of p,d,q for seasonal & non seasonal arima for a customer and pick the one with least MAPE etc but doing it may not give the most accurate result.

Is there any better way of approaching this problem? Or are there any better methods to tackle the scale of this problem were i don't have to build 3000 arima models instead build fewer models?

Note: Getting a customer level forecast is must I cannot group customers and forecast.

Improve this question
$\endgroup$
5
  • 5
    $\begingroup$ A key question is whether the different customers have something in common. If there are groups of customers that share common features, something like a panel data approach could be useful. Some of the parameters would be individual for each customer and some would be common. For the common parameters, they would be estimated more efficiently in the common model than having 3000 individual models. Regarding the scale of the model, writing code for 3000 models is not more difficult than writing it for 30 models, and ARIMA estimation of not-too-long series with not-too-long lags is quite fast. $\endgroup$
    – Richard Hardy
    Commented Apr 11, 2015 at 13:31
  • 1
    $\begingroup$ You might want to look at the automated procedure implemented in the R forecasting package, and described in Hyndman and Athana­sopou­los's book. You may be able to use that R module directly OR at least see how they do the search. otexts.org/fpp/8/7 $\endgroup$
    – zbicyclist
    Commented Apr 12, 2015 at 5:35
  • 1
    $\begingroup$ If we were approaching this problem 50 years ago, we'd probably use an exponential smoothing approach, just to keep the calculation resource requirements under control. A model like damped trend exponential smoothing might be something to look at. Here's an introductory set of slides: bauer.uh.edu/gardner/presentation.asp $\endgroup$
    – zbicyclist
    Commented Apr 12, 2015 at 5:41
  • $\begingroup$ Thanks @Richard Hardy & zbicyclist. Can you tell me which algorithm can model the common factors & individual factors? If i use auto.arima in R it can only run non seasonal arima & not seasonal arima right? Is there any particular forecasting algorithm which can run on the entire 3k customers are give accurate individual forecast for each of the 3k customers? $\endgroup$
    – GeorgeOfTheRF
    Commented Apr 13, 2015 at 11:22
  • $\begingroup$ Just thinking - with 3k customers you would surely not see the same times of purchases for each? How did you set the time scale? Also, besides some sort of id, what info do you have (eg purchase "type", whether they are new/repeat customers, location, etc. $\endgroup$
    – probabilityislogic
    Commented Mar 29, 2018 at 9:55

3 Answers 3

Reset to default
2
$\begingroup$

You certainly don't want to build 3,000 separate models! Not only would that be computationally and administratively cumbersome, but it would also mean that each model only has data from one customer, and so you are ignoring data from other customers in each model. This would effectively mean that your predictions for a customer are based solely on their individual (small) dataset, and you are not using the (large) dataset from other customers to help with any aspect of your prediction.

A much better approach than that monstrosity would be to formulate some kind of general time-series model that has an effect term for each customer. There are many ways this could be done, and the ultimate test is to see what model fits your data well, and makes good out-of-sample predictions. Here is an example of a simple model to get you started thinking about the possibilities.

An example model: If you let $X_{i,t}$ be the log-revenue for customer $i$ at time $t$ you could formulate a simple Gaussian ARIMA model including customer-level mean and variance effects as:

$$\phi(B) \Delta^d (X_{i,t} - \mu_i) = \theta(B) \sigma_i \varepsilon_t \quad \quad \quad \varepsilon_t \sim \text{IID N}(0,1),$$

where the AR and MA characteristic polynomials are:

$$\phi(B) = 1 - \phi_1 B - ... - \phi_p B^p \quad \quad \quad \theta(B) = 1 + \theta_1 B + ... + \theta_q B^q.$$

As you can see, this is a standard Gaussian ARIMA model, but with each series varying with a different mean and variance parameter, for each customer. Once you have used the data to estimate the parameters you could then make predictions for an individual customer based on their estimated mean and variance in the series. Some customers give you more revenue, so they will have a higher mean. Some customers vary their revenue more, so they will have a higher variance. Nevertheless, other aspects of the model are estimated using the data from all the customers.

It is important to note that there are many variations you could make to this model, such as using customer-level random effects, or a hidden-state process with another time-series process for the underlying mean for each customer. Really, there are all sorts of variations you could make, and you will need to see what fits your data. However, this kind of model has the advantage that all the data is used simultaneously to estimate the parameters, so the prediction for an individual customer still depends on all the data.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks. Can you explain the paramters B,d etc used in the equation? I understand Arima model but what do u mean by Gaussian ARIMA model? Any links or pointers on the same will be helpful!. $\endgroup$
    – GeorgeOfTheRF
    Commented Jul 5, 2018 at 9:01
  • $\begingroup$ Well, you mentioned in your question that you are considering a seasonal or unseasonal ARIMA($p,d,q$) model, so $d$ is just the parameter of that model you mentioned (i.e., it is the order of the differencing). The operation $B$ is the backshift operator, which is the most common way to express this kind of model. (It is sometimes written alternatively as $L$ for lag operator, but that is the same thing.) When we say that a model is "Gaussian", that simply means that the errors have a Gaussian distribution (i.e., the normal distribution). $\endgroup$
    – Ben
    Commented Jul 5, 2018 at 10:09
1
$\begingroup$

I faced very similar task at work. First I used an automated ARIMA function applied to each timeseries separately. It worked sufficiently fast for my purposes.

Then I studied the properties of timeseries by doing a big comparative analysis, using both ARIMA, and a number of simpler techniques like linear models with preprocessed inputs, random walk, random walk with a drift.

I found that simpler models work tens times faster, but what comes to how well they work totally depends on your data. In my case, most timeseries were not distinguishable from random walk, so using the last value (or mean) as a forecast looked sane.

Try linear models, they can be surprisingly fast and accurate.

Improve this answer
$\endgroup$
0
$\begingroup$

I had a similar project where forecasting was required for 3000 Distributors for one of my clients. I used Automated ARIMA and looped it through clusters of Distributors by their type.

I had to finally formulate a parallel package to handle multiple Distributor types in parallel in order to increase efficiency as models needed to be trained once a month. It can be easily done using Parallel packages in R and Rob Hyndsight Forecast package.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.