Lift Analysis: Promotion Imbalanced data

Question

I have been reading about imbalanced dataset lately and new to data science Domain. I came across this article which talks about various methods to solve this problem in R. Considering e-commerce data, we have Baseline sales(sales without promotion) and Incremental sales(sales with promotion). As this will be a true example of the imbalanced dataset, Supposedly in the hypothetical example, we have 2 million sales record for baselines for a year and 100K sales record for incremental sales. We can't Undersample and oversample as we will loose important information and won't be able to handle a large date with duplicates because of oversampling. If we want to tent the significance of promotion we can use SS3 ANNOVA but we will not have a true reflection. How do we find the lift of promotion or incremental sales over baseline sales in this scenario as CSL(cost sensitive learning) will not balance the data well. Any thoughts on how to handle this scenario? This lift analysis would help to analyze if promotion significantly helped in getting more sales FYI.

And regarding seasonality for year-long data, we can do seasonal adjustment using time decomposition in R and subtracting/dividing based on series.

Answer 1

Assuming that the two populations have roughly the same and pretty much uniform characteristics (demographics basically) and you want to test the difference of a numerical variable (revenue gained or ROI or whatever) what I would do if I were you is:

1. Split the dataset to train and test sets at random 1.4M baseline and 70K exposed to advertising training set and the rest 0.6M and 30K test set

2. Split the training set into many smaller datasets (with replacement) consisting of a balanced 50-50 sample from both exposed and non exposed to advertising.

3. Use cross validation to tune the model's parameters (you can try many different kinds of models here) to predicting the difference between your response variable in the two groups.

4. To assess how accurate your model is split the testing set dataset again into equal advertised and non advertised people but this time without replacement in the non-advertised population and the same people every time in the advertised population (i'm open to different approaches here) and see how well your model predicts the difference between the two populations in these (unseen) data.

5. Re iterate the above process from step 1, changing the seed every time, and report the average error (difference between predicted and actual value)