I'm having a hard time wrapping my head around experiment design and what appears to be a disconnect to experiment outcome analysis.
If I am proposing a z test of two independent proportions (conversion rates) where I have an assumed historic conversion rate of p2 = 0.10 and I want to determine the required sample size to detect an effect of 20% (p1 = 0.12) using a one tailed test with alpha = 0.05 and power = 0.80 I would arrive at required sample sizes of roughly 3,025. I am comfortable with this calculation utilizing cohen's h and have confirmed using G*Power as well.
However, If I were to fast forward and assume that I conduct the experiment and indeed my control achieves the assumed historic conversion rate of 0.10 and my treatment achieves my hypothesized conversion rate of 0.12 and I check the p value for that experiment I receive p = 0.006.
Why is this? This makes it seem like I am vastly over sizing my required sample sizes. Am I seeing this effect because my sample sizes are ultimately being inflated to protect against type II errors with 80% confidence?