Verbiage on rejecting null hypothesis, Sample Size, testing

Question

I'm new to using statistics in psychology, and I'd like to seek some guidance from this forum.

Context: I'm running an experiment involving 5,000 participants. Half of them (2500) will receive a promotional email with a coupon. The other half will not. I want to evaluate sign-up rates under both conditions.

My research question is: Will the email with the coupon significantly improve sign-up rates, relative to the email without the coupon?

H0: The email with the coupon will not significantly improve sign-up rates.

Mathematically:

H0: Signup (Coupon) - Signup (No Coupon) = 0

Question 1

1. Is it 'legal' to state that: "To reject H0 and claim that the coupon significantly increases sign-up rates, the difference in sign-ups must be significantly different from 0"?

I'm specifically concerned about the phrase "claim that the coupon significantly increases sign-up rates". I want to write this because it makes sense to a layperson reading the report, but I also understand that in hypothesis testing, we can only state that we either reject or do not reject the null, so this phrase would sound like I'm "accepting the alternative" which is inappropriate.

Question 2

1. Is there any issue with false positive, given that my sample size is quite large, and what can I do about it?

I understand that in large sample sizes, small effects can get magnified to the point of being statistically significant. In my instance, I have 2,500 observations each for treatment and control, and I'm essentially testing whether the difference e.g. 200 is different from 0. Intuitively it sounds like I'm definitely going to get a significant result, so I was wondering whether I need to do anything differently (use percentage differences instead) to avoid Errors.

Question 3

1. Should I be using a two-tailed t-test, or something else? While we were taught t-tests, I understand that it's more about comparing the means between two groups. In this case, I'm comparing absolute differences.

Thanks for your help in advance.

Answer 1

Regarding question 1, the use of the phrase "significantly increases" is a but dubious. You can reject the null and have the promotion increase sign up rate by a negligible amount (say half a percent). Under some circumstances, that is not a significant increase in the sign up, and yet the null is rejected.

Its hard to put it in a phrase, but I might say

If the difference in sign-up rates is larger than what sampling error could produce under the assumption that the coupon has no effect, we would consider the result "statistically significant" and conclude that the coupon does have some effect, the size of which we would estimate from the data.

Regarding question 2, it depends on the effect size you're interested in measuring. You should definitely examine the rate of response rather than the counts, that much is true. If you have an idea of what the sign up rate is for the control group, you can do a power analysis. A lot has been written on that topic on this website.

Regarding question 3, use a test of proportions. Additionally, you can compile your results into a contingency table and use any one of these tests I outline here.