I'm new to using statistics in psychology, and I'd like to seek some guidance from this forum.
Context: I'm running an experiment involving 5,000 participants. Half of them (2500) will receive a promotional email with a coupon. The other half will not. I want to evaluate sign-up rates under both conditions.
My research question is: Will the email with the coupon significantly improve sign-up rates, relative to the email without the coupon?
H0: The email with the coupon will not significantly improve sign-up rates.
H0: Signup (Coupon) - Signup (No Coupon) = 0
- Is it 'legal' to state that: "To reject H0 and claim that the coupon significantly increases sign-up rates, the difference in sign-ups must be significantly different from 0"?
I'm specifically concerned about the phrase "claim that the coupon significantly increases sign-up rates". I want to write this because it makes sense to a layperson reading the report, but I also understand that in hypothesis testing, we can only state that we either reject or do not reject the null, so this phrase would sound like I'm "accepting the alternative" which is inappropriate.
- Is there any issue with false positive, given that my sample size is quite large, and what can I do about it?
I understand that in large sample sizes, small effects can get magnified to the point of being statistically significant. In my instance, I have 2,500 observations each for treatment and control, and I'm essentially testing whether the difference e.g. 200 is different from 0. Intuitively it sounds like I'm definitely going to get a significant result, so I was wondering whether I need to do anything differently (use percentage differences instead) to avoid Errors.
- Should I be using a two-tailed t-test, or something else? While we were taught t-tests, I understand that it's more about comparing the means between two groups. In this case, I'm comparing absolute differences.
Thanks for your help in advance.