0
$\begingroup$

I'm relatively new to statistical testing and was hoping for some guidance.

I have a data set containing two groups.

Group 1: website users and their number of logins for May & June 2018
Group 2: website users and their number of logins for May & June 2019

My website ran a promotion during May and June 2019. I'd like to test to see if my promotion had increased the number of logins into my website. The problem is that the number of users in 2019 is higher than they were in 2018. I'm afraid this may skew my statistical testing and incorrectly conclude that the promotion did increase website logins.

My initial thought was to use t-testing for dependent samples, but the users in my data set are not the same in both groups(new users in 2019). Same goes for independent sample testing, but rather my groups contain the same users from 2018 along with new users fro 2019.

Should I pull users that exist in both groups and perform a t-test for dependent samples?

Any additional insight is greatly appreciated.

Improve this question
$\endgroup$
1
  • $\begingroup$ Is it possible to use the overall difference between 2018 and 2019? Maybe call that $\delta?$ Then test $H_0: \mu_{19} = \mu_{18}+\delta$ vs. $H_a: \mu_{19} > \mu_{18}+\delta.$ $\endgroup$
    – BruceET
    Jul 24, 2019 at 4:31

1 Answer 1

Reset to default
0
$\begingroup$

I think it would be helpful if we tried to clarify your statistical question, rather than the marketing question. Paraphrasing from your post the statistical question would be: is there a true difference in the number of logins in 2018 vs number of logins in 2019?

There are a number ways to answer this question. There are some longitudinal regression methods that could be used, but I think the easiest would be what you proposed: compare logins among users who exist in both the 2018 and 2019 dataset. Statistically, the login records of the users in the 2018 year should act as their own controls for the 2019/promotion year. In this case a matched pair t-test with the user number/identifier would be appropriate.

The caveats are (1) the new 2019 users are not considered meaning you assume they are similar to 2018 users, and (2) this assumes time is not a confounder between users and number of logins (that users' login behavior is not changing because of time passing) (3) the promotion is not explicitly modeled here so it will only be an associated with logins.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the feedback! I ended up going with the paired t-test for users that existed in both sets. $\endgroup$
    – DieHard345
    Jul 26, 2019 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.