How to test for significant difference between 2 linear regression models?

Question

The following scatterplot represents the Number of Users in a website against Number of Day.

After the first month, on day number 31, a campaign was launched and the users went slightly up. I fitted a linear regression model for the first month, and then a second one for the second month.

So my questions are:

1. Is this uplift the result of a better campaign or is the result of the already existing trend? In other words, is the difference between the two slopes significantly different or not? What about the intercepts? How can I compare the 2 models?

2. What is a big enough sample to fit the models before and after the campaign launch?

3. Are there any other methods besides linear regression models? Maybe a T-test or an Anova model?

For anyone that wants to reproduce the scatterplot and the models, the dataset I used was this one:

Day_Number Users Campaign 1 114 0 2 151 0 3 155 0 4 157 0 5 143 0 6 188 0 7 143 0 8 181 0 9 224 0 10 155 0 11 223 0 12 247 0 13 210 0 14 184 0 15 231 0 16 255 0 17 292 0 18 245 0 19 254 0 20 246 0 21 343 0 22 329 0 23 284 0 24 287 0 25 338 0 26 341 0 27 352 0 28 358 0 29 350 0 30 362 0 31 503 1 32 582 1 33 524 1 34 400 1 35 285 1 36 559 1 37 648 1 38 392 1 39 642 1 40 665 1 41 631 1 42 789 1 43 459 1 44 625 1 45 586 1 46 854 1 47 818 1 48 670 1 49 594 1 50 672 1 51 919 1 52 900 1 53 960 1 54 899 1 55 1046 1 56 901 1 57 759 1 58 813 1 59 923 1 60 887 1

Answer 1

Are you currently using Campaign in your model, or are you just using it as a way to divide your data to fit two models? Right now, it looks like you are fitting two separate versions of

\begin{equation} \text{Users} = \beta_0 + \beta_\text{1}\text{Day} \end{equation}

Testing for differences in the models shouldn't be difficult if you nest them. If you use Campaign as a second predictor in your model, you can determine whether there is a significant difference in the intercepts based on whether Campaign is significant. Because Campaign is 0 or 1, that term will be an additional constant that is added depending on whether the campaign is happening or not.

\begin{equation} \text{Users} = (\beta_0 + \beta_2\text{Campaign})+ \beta_\text{1}\text{Day} \end{equation}

If you add a Campaign x Day interaction, then that can tell you whether there is a significant difference in your slopes. Similar to the equation above, when the campaign is happening, your slope will change from beta_1 to (beta_1 + beta_12)

\begin{equation} \text{Users} = (\beta_0 + \beta_2\text{Campaign}) + (\beta_\text{1}+ \beta_{12}\text{Campaign})\cdot\text{Day} \end{equation}