I have a very large marketing data set (300GB in size) and I am wondering what ML models I can apply to it. The dataset consists of transactional data from supermarkets. The type of data I have is the following:

  • There are 20 years worth of data, with 52 weeks in each year.
  • 50 markets/regions -- Within each region there are approximately 20-40 stores. ---Each store sells approximately 40 different products (think; shampoo, toothpaste, chips, frozen pizza, beer etc.) ---- Each product consists of many brands ---- Each product has many characteristics

  • For a few markets I have Panel level information where I can determine income, race, sex etc. (approx 1 million observations).

For the whole data set I estimate that there are approx 5-10 billion observations.

So I have 20 year time series transactional data for 50 different markets, with a lot of information on the products (type of packaging, Price, qty sold, glass bottle or tin can, colour, flavor etc.

What I have currently done:

  • Apply a simple LSTM model to forecaste future sales for certain brands
  • Apply XGBoost in order to classify which customers Will purchase a certain product

What I want:

  • I am looking also for some more "economic" theory to be applied. It is great to apply ML in order to forecast sales etc. but I want to prove/disprove some economic theory. i.e. are certain customers (based on PANEL characteristics) more willing to buy a particular Brand than other customers. Do customers who shop at stores with fewer choice choose the same product as customers who shop at stores with a higher selection.


Could it be possible to apply reinforcement learning over a sequence (time) in this model?

At the moment I am just brainstorming some ideas so your advice, comments Will only add to future ideas.

My question is: Given this data set which ML model would you apply and to which problem?

I have some ideas myself but I want to know if there are better suggestions for such a data set.


closed as too broad by rolando2, Michael Chernick, kjetil b halvorsen, Ferdi, jbowman Jan 22 at 5:15

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.