3
$\begingroup$

I am looking for references regarding the forecasting of the demand for multiple products in an automated manner (no human intervention). Specifically, I have a number of clustered products for which I am trying to predict the future (daily) demand for a price using historical data. I have tried using a traditional time series approach, however I need to forecast the demand for each price level which makes my data a multivariate time series.

Is a VAR model the most commonly used approach? I guess the different points are highly correlated (i.e. demand for day $t$ at price \$1 $<$ demand for day $t$ at price \$2 $<$ demand for day $t$ at price \$3). Another (I guess naive) approach would probably be to have a forecast for one price level only (I have no idea how to select a point in the demand curve) and then learn the quantity proportions for the remaining prices.

I know that I need to try different methods and explore the data but my main issue is that the volume of data is huge and changes a lot so I cannot re-estimate my model manually.

I am still a beginner, so apologies if this is a straightforward question.

Improve this question
$\endgroup$
4
  • $\begingroup$ You likely have only one price per time point (or per customer), so you don't necessarily have multivariate time series, but causal ones. You could try running a regression on price with ARMA errors, using auto.arima in the forecast package for R, feeding prices into the xreg parameter, possibly after some transformation. It's a bit unclear to me where the multi-product aspects come in. Do you assume some total demand that breaks down among the different products? $\endgroup$
    – Stephan Kolassa
    Commented Nov 10, 2015 at 15:27
  • $\begingroup$ Thank you for the reply. In my setting, I can always see the quantities sold for all prices, i.e. I have a curve (p^t_1,q^t_1), ..., (p^t_n,q^t_n) where (p^t_i, q^t_i) are the price-quantity pairs for day t (assume p^t_i < p^t_j for i<j) and I want to forecast the full curve for the next day. The multi-product aspect relates to the underlying population of products which is not fixed (I do observe each product's attributes) $\endgroup$
    – user90772
    Commented Nov 10, 2015 at 15:33
  • $\begingroup$ When you write the volume of data is "huge," how many products are you referring to? How often are you making these forecasts? Is it for actual product pricing, e.g., for online sales or for financial/strategic planning? How far ahead do you need to make your forecasts? Is the unit of analysis the individual sales transaction or has it been aggregated up to product sales per, e.g., hour, day or week? How large or wide is the historic time period available? $\endgroup$
    – user78229
    Commented Nov 10, 2015 at 17:03
  • $\begingroup$ Thank you for the questions. There are thousands of different product variations (some are similar but might be different in some attributes such as colour or location), the forecasts are to be made daily for the next 7 days (the full planning horizon is approximately a month but I would not manage to have any meaningful prediction for that long). Sales have been aggregated per day (raw data are per second) and I am using historical data of approximately a month (could be a couple of months in some cases or 2-3 weeks in other cases). $\endgroup$
    – user90772
    Commented Nov 10, 2015 at 17:11

2 Answers 2

Reset to default
1
$\begingroup$

That you have daily aggregated sales information for only 1 or 2 months, even for thousands of products and their variations, limits the possible analyses. For instance, if your sales are strongly seasonal, e.g., as a function of the winter holidays or conversely the warm summer months, then you won't be able to integrate this potentially important information into the model.

Per your question, as I see it, you have two broad options: turn the process over to one of the many vendors out there providing automated retail solutions or, alternatively, do it yourself.

Regardless of which option you choose, you would be wise to do a whole lot of exploratory work on this dataset just so that you feel like you understand it. That way, if the solution used (whatever it is) returns nonsense, you will have a good sense of when that is occurring.

There are plenty of vendors of turnkey, automated retail solutions. Here are a few of the big names. Others can provide additional names:

IBM's Demandtec or omni-channel solutions http://www-01.ibm.com/software/info/demandtec/

McKinsey's Periscope for retailers http://www.periscope-solutions.com/

Planet Retail http://www1.planetretail.net/what-we-do

Khi Metrics http://www.groceryretailonline.com/doc/khi-metrics-0001

And in terms of DIY, given the massive volume of information and the relatively short time frame (~30 days), I doubt that traditional, univariate, "Box-Jenkins," ARIMA, VAR-type approaches lend themselves that readily to turnkey solutions. First of all, the approaches rely on many more data points just to initialize the lags and moving averages than 30 days. Second, and to the best of my knowledge, they aren't fully multivariate in the sense that a pooled or multilevel model might but others can disagree. Regarding the suggestion made to use Hyndman's functional time series analysis, I can't evaluate the adequacy of that recommendation.

I think you need to find a functional form for the model that is flexibly appropriate for:

1) The relatively short span of historic information

2) The massively categorical nature of the products

3) The hardware and software challenges of processing huge volumes of information

4) The need to update and produce compiled parameters on some regularly scheduled basis so that automated predictions can be made

Presumably -- or better, hopefully -- you aren't doing this work on a single laptop or workstation but have access to some sort of multi-core platform such as AWS integrated with software such as Ufora. Ufora offers massively parallel analyses on AWS. There are workarounds to the limits, even in the cloud, of RAM or working memory for statistical modeling. These include the many variants of so-called "divide and conquer" algorithms which, essentially, amount to a greatly extended random forests approach. I've heard of shops where they will execute 3 or 4 million "mini-models" or random forest resampling iterations in a few hours on a 100 core Hadoop platform, then roll it up on the back end. A good reference for this is Chen and Xie's http://dimacs.rutgers.edu/TechnicalReports/TechReports/2012/2012-01.pdf

In terms of the functional form, I think you're looking at a variation of multi-level modeling...whether that be pooled OLS, HLMs, GAMS, whatever, is a process that needs to be explored and cannot be determined in advance.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thank you for the extensive answer! There is indeed a hadoop cluster where data can be analyzed and I am looking for a DIY solution. Regarding the multi-level modelling, I guess this is about learning about the different sub-populations of products, as a hierarchical time series, or am I wrong here? There is one extra dimension, that of a price, for each quantity I would like to make predictions of. So, I have e.g. <price, colour, location> tuples for a population of similar products and I want to learn the aggregate quantity at each price for all of them. Sorry if this comment seems naive. $\endgroup$
    – user90772
    Commented Nov 11, 2015 at 9:37
  • $\begingroup$ What I would essentially want as an output is the daily CDF of the prices for the whole population of products plus a forecast of the actual quantity at the maximum price. $\endgroup$
    – user90772
    Commented Nov 11, 2015 at 10:09
  • $\begingroup$ Rather than just using the maximum price, why not build a demand curve based on the elasticity of demand, given price and controlling for the other factors in the model? That's an optimizable function. $\endgroup$
    – user78229
    Commented Nov 12, 2015 at 12:05
  • $\begingroup$ Are you suggesting using some parametric form for the demand curves if I understand correctly? Otherwise, the elasticity of demand is still a curve to be estimated. The forecasts are to be used in an optimization system later based on other factors. $\endgroup$
    – user90772
    Commented Nov 12, 2015 at 12:24
  • $\begingroup$ That is what I'm suggesting. The actual methodology for several functional forms for demand are developed in Lee Cooper's book, referenced above. $\endgroup$
    – user78229
    Commented Nov 12, 2015 at 12:27
1
$\begingroup$

This looks tricky. You may want to think about how you manage to sell the same product at different prices - this may give you an idea of how to segment your problem and/or model it causally.

You may be able to forecast total sales per product across all prices and then disentangling them by price. Look at Forecasting functional time series by Hyndman & Shang for a possible approach. They forecast age pyramids over time. Total population would correspond to total sales in your example, and ages to prices.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for the helpful answer. There are a lot of product attributes but I will give it a try! $\endgroup$
    – user90772
    Commented Nov 10, 2015 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.