I have a very large marketing data set (300GB in size) and I am wondering what ML models I can apply to it. The dataset consists of transactional data from supermarkets. The type of data I have is the following:
- There are 20 years worth of data, with 52 weeks in each year.
50 markets/regions -- Within each region there are approximately 20-40 stores. ---Each store sells approximately 40 different products (think; shampoo, toothpaste, chips, frozen pizza, beer etc.) ---- Each product consists of many brands ---- Each product has many characteristics
For a few markets I have Panel level information where I can determine income, race, sex etc. (approx 1 million observations).
For the whole data set I estimate that there are approx 5-10 billion observations.
So I have 20 year time series transactional data for 50 different markets, with a lot of information on the products (type of packaging, Price, qty sold, glass bottle or tin can, colour, flavor etc.
What I have currently done:
- Apply a simple LSTM model to forecaste future sales for certain brands
- Apply XGBoost in order to classify which customers Will purchase a certain product
What I want:
- I am looking also for some more "economic" theory to be applied. It is great to apply ML in order to forecast sales etc. but I want to prove/disprove some economic theory. i.e. are certain customers (based on PANEL characteristics) more willing to buy a particular Brand than other customers. Do customers who shop at stores with fewer choice choose the same product as customers who shop at stores with a higher selection.
Could it be possible to apply reinforcement learning over a sequence (time) in this model?
At the moment I am just brainstorming some ideas so your advice, comments Will only add to future ideas.
My question is: Given this data set which ML model would you apply and to which problem?
I have some ideas myself but I want to know if there are better suggestions for such a data set.