# Statistical tests for knowing the effect of a campaign on sales

I am currently working on my very first real life Data Science problem and I am facing a bit of a challenge in formulating the solution.

The question is to find out if conducting a campaign has an effect on the sales overall. To do this, we have the data related to customer information, sales invoices, campaigns conducted with reference to the customers.

I thought of the following ways to tackle this:

• We could simply split the customer base into 2 groups, those participating in campaigns and those not, and then just perform a statistical test (not sure which one yet) to see if there is a significant difference in their aggregate sales.
• Instead of comparing the aggregate sales, we could also compare just the frequency/count of sales across the 2 groups (not sure which model to use here)
• We could go for a logistic regression approach where the target variable represents whether a sale was made or not and then check which coefficients had the highest effect on the target

I am not completely sure as to what would be the best approach here. It would be really helpful if I could get some insights into that.

If there are better approaches to tackle this problem, please do share them! I would love to know more about them.

Edit: I would like to add more information regarding the data itself in case it makes a difference in the assumptions made.

I have 3 datasets

• Customers: This table, as expected, contains anonymized information about the customer, what campaign they were subjected to etc.
• Campaign: Information about the campaigns conducted like start and end dates, some descriptions regarding the campaign etc.
• Sales: This contains information about date of sale, the value of purchase, location of sale, etc.
• Does your dataset contain all customers, or just the ones where sales were made? Can a customer have more than one sale? Was all data collected over more or less the same period of time, or were there customers that spent a lot longer being exposed (or not) to the campaign(s)? When you say 'aggregate sales', you mean total amount spent versus option 2/3 where it's just a binary 'sale made' indicator (or cumulative count)? Oct 18, 2023 at 12:39
• It contains all customers and yes a customer can have more than one sale. The data was collected over 1 year and I am not entirely sure about the exposure part. Yes that is exactly what I meant when I said "aggregate sales". Oct 18, 2023 at 12:48

Based on your posts and comments, I imagine your data on two levels. You have a binary outcome, sales (made or not), let’s call it $$Y$$. As you mentioned customers can have more than one sale. You also have information on customers with no sales. And I think there may be customers who made purchases sometimes but not in others. So, there is a nesting factor which I’ll call $$\operatorname{ID}$$. You might have predictors at the sales level, denoted as $$X$$ (maybe the date of sale; some sales could be on holiday season, etc.), and customer level, denoted as $$Z$$ (e.g., sociodemographic variables). I also assume that participation in a campaign is at the customer level (someone participated or not).

I would start with a random-intercept binary logistic regression model, taking into account this nested structure:

\begin{aligned} \operatorname{Y}_{i} &\sim \operatorname{Binomial}(n = 1, \operatorname{prob}_{\operatorname{y} = 1} = \widehat{P}) \\ \log\left[\frac{\hat{P}}{1 - \hat{P}} \right] &=\begin{aligned} &\alpha_{j[i]} + \beta_{1}{X_1} + \cdots+ \beta_{n}{X_n} \end{aligned} \\ \alpha_{j} &\sim N \left(\gamma_{0}^{\alpha} + \gamma_{1}^{\alpha}Z_1 + \cdots+ \gamma_{n}^{\alpha}{Z_n}, \sigma^2_{\alpha_{j}} \right) \text{, for ID j = 1,} \dots \text{,J} \end{aligned}

You can expand this model based on your analytical interests. As I said, I assume that participation in a campaign is at the customer level but that might not be the case, hence the third question raised by PBulls. Some sales could be made before a customer participated in a campaign or some customers could be exposed to campaign longer than others. If that’s the case, the model changes, and you should interpret the results accordingly.

• Thanks a lot for the answer! I am glad to see that I was at least on the right track. Your assumption regarding the customer participation being at customer level holds true in my case. The tricky part would be to include the exposure time period in the model. For that I am thinking about creating a variable that records the delta in exposure to see if that affects the results in any way. One thing I would like to know is, if I just plug all of the variables into the logistic regression model, what kind of a difference would it make? Oct 30, 2023 at 8:15
• I am glad I could help. If you plug all variables into logistic regression, the model will run (if there is no other issue) and you will get the results but it will not take into account the nested structure in the data (e.g., no random effects for ID). Without knowing more about your analysis, it's hard to tell specifically what difference it would make. You can always compare this model with the multilevel model, and see how the results differ. Oct 31, 2023 at 7:15
• I added some more description about the data itself in case it would be helpful. I am still thinking about the nested structure of the data. If we consider that the customer information introduces its own random variable all of this makes complete sense. However, I dont fully understand why that would be the case. If they are in fact introducing some randomness, wouldnt the model with all the variables account for it? Why would we explicitly introduce another variable? Nov 1, 2023 at 10:33
• How would you account for the fact that there are several observations from the same customer (as we assume)? Are you going to include ID as a fixed effect? The issue is not randomness (I guess here we mean different things). There is violation of the assumption of independence. Nov 1, 2023 at 15:06
• I understand now. I forgot about the assumptions underlying the logistic regression model. Apologies for that and thanks again for explaining! Nov 2, 2023 at 7:38

After researching the topic a bit better, I have come up with the following solutions which make the most sense to me.

1. A Chi-Squared Test to compare the count data for the 2 groups (sale, no sale) would be good. Reference here.

2. Basing the conclusion solely on the Chi-Squared test would not be a very strong foundation and hence conducting logistic regression with the variables available would be helpful.

Combining both of these methods should give the result in a good way so that actual decisions can be taken.

If someone else has better answers I would really appreciate it!

• It looks like you have information on two levels: sales and customers, and the former is nested in the latter. I would have used a modeling strategy taking this structure into account. Pbulls' questions are related to this issue. Oct 21, 2023 at 20:18
• @T.E.G Could you elaborate on the modelling strategies you have in mind? I am still working on this problem so it would be nice to know. Oct 23, 2023 at 8:06