I have been told that I need both significance and power for my AB results to be valid. I researched a lot for this and the above statement is not making sense. I get that we need high enough power to not reject the null hypothesis and assuming that the new feature has bought no actual effect, but why do we need power to reject the null hypothesis when my confidence interval is already so high?
My confusion is as below:
Power is (1-Beta). So higher the power, lower the probability of type 2 error (not rejecting the null hypothesis when it is false). The thing is, I am rejecting the null hypothesis as my results are very significant and alpha is already low.
Lower the alpha, more the sample size required at the same power: This further adds to my belief that you don't need statistical power to reject the null hypothesis. I mean, are we really saying the more my confidence interval, the more data size i will need to validate the effect?
I am not sure if I am missing some key concept. Please help me out as I am pretty sure that the new feature has positive conversion and I have already reached 99.99% CI.