I have two years of sales data on a daily basis of 2,000 products. I am not really interested in Time Series modelling but rather on some sort of count data model. The issue is that sales are really sensitive to promotions so it's difficult to model them.

In the following plot you can see how it looks like (sorry for the axis, I can't disclose what company and quantities it is about).


So sales are quite steady but when there is a promotion (e.g. 50% discount) sales increase a lot in just a single day. I have tried with a model such that:

$Sales=\beta_{0}+\beta_{1}DayWeekAverage + \beta_{2}MonthAverage + \beta_{3}Promotion + \beta_{4}Timetrend + \varepsilon$

Where WeekAverage is a factor variable (dummies) for each day of the week, and MonthAverage for each month. The problem of this model is that it's very simple and only manages to model 1 product at a time.

I think a panel structure to model the 2,000 products may be good enough because there is heterogeneity in te effects between products. I have thought on a Generalized Linear Mixed Model, but I'm not sure if this may do the trick. Also a Random Effects model may be good enough.

What do you suggest? What other models would be good to model this kind of data?

  • 1
    $\begingroup$ You can also consider clustering products to create cluster level random-effects. The choice for clustering algorithm would depend on the type of products (e.g. hierarchical or not). This would also help products where sales are sparse. Making one model per product would not be optimal, but there sure would be products which would benefit a lot from it (typically high sales products). Therefore a mix of both strategies should give you near-optimal results. $\endgroup$ – Ujjwal Kumar Jan 10 '17 at 10:46
  • 1
    $\begingroup$ @UjjwalKumar But how do you suggest to cluster the data? I would like something more empirical in the sense don't want to do naïve groups. Some sort of K-Means of clustering algorithm? Not very sure of this approach though... $\endgroup$ – adrian1121 Jan 10 '17 at 10:54
  • 1
    $\begingroup$ I'm currently working on a dataset which is pretty similar to yours. First of all: what kind of products are you trying to model? Are those from a single retailer? (Just to know how much granular and indeed noisy your ts could be). I'm using a univariate poisson GLM model, but I include promotions and stockouts of other products as regressors. That's definitely a start. Unfortunately there's no multivariate poisson distribution, so the only way to capture the covariance structure of the product could be in a hierarchical model with shared hyperparameters. $\endgroup$ – Tommaso Guerrini Jan 10 '17 at 11:05
  • 2
    $\begingroup$ If you are interested in the least here's a post of mine of some months ago with R code: stats.stackexchange.com/questions/244660/… By the way, since I'm still working on this, I leave you my mail if you wish to share ideas: guerrinitom@gmail.com $\endgroup$ – Tommaso Guerrini Jan 10 '17 at 11:06
  • 1
    $\begingroup$ @UjjwalKumar I don't suggest clustering the products sales directly, you could try to group the residuals of your univariate fitted models. Personally I used the product category (I had beverages from Wine to Energy drinks to Water and whatever) and I extracted the brand from the strings as handmade clusters. $\endgroup$ – Tommaso Guerrini Jan 10 '17 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.