Designing Frequentist A/B Experiments What parameters should one pre-specify before designing/running an experiment? (e.g. a two-sample, one-side, t-test)
My understanding is that on establishes: 


*

*Null hypothesis

*Maximum Significance level

*Minimum Power

*Minimum Effect size


Is the choice of maximum/minimum in the lines above correct?
Is it correct to say that one needs to establish all of the above to then estimate the required sample size? Or are there any other free parameters that one needs to establish first? 
 A: There are two questions here:


*

*What should one establish before an experiment?

*What is the relationship between $\alpha$, $\beta$, effect size and study size?


The two are separate, because you can not presuppose that every study requires a Neyman-Pearson'ian frequentist perspective. In many statistical frameworks, terms such as Power or Significance levels are not employed as by N&P. For example, within certain Bayesian parameter estimation frameworks, it is valid to not pre-specify your sample size, but sample until the results are conclusive.
So 1 and 2 are truly separate questions unless you focus on frequentist NP analyses (and there are even frequentists who hate power analyses!).
However, if, as your title specifies, you decide on the NP framework, in a power analysis, one parameter can float when all others are fixed. So you can use the relation between these terms to calculate the necessary minimal size to guarantee your $\alpha$ and $\beta$ for your assumed effect size. (I'm phrasing it here as $\alpha$ and $\beta$ rather than Power and Significance level because I think thinking about what's maximal versus minimal is easier this way.) 
One typical usage would be:


*

*Specify your $\alpha$ level (maximal false positive rate); basically, significance level

*Specify your $\beta$ level (maximal false negative rate); note the inverse relationship between Power and $\beta$

*Specify a minimal effect size; this is basically equivalent to setting up rejection regions for your hypotheses

*These yield the minimal n for setting up experiments so that out of 100 of these experiments, if H0 is true, on average no more than $\alpha$ would reject H0, and if H0 is false to at least the extent specified by your effect size, on average no less than $\beta$ would reject H0.


If any of these parameters changes, the others change with it. If you decrease the sample size and keep $\alpha$ fixed, power to detect an effect of the specific size gets worse, and so on.
If your sample is larger, you will either have a higher power for detecting effects of the specified size, or the same power for detecting (slightly) smaller effects, and so on.
For example, for these experiments where a true effect is stronger than the one specified for the power analysis,  the actual false negative rate within these experiments will be better than their calculated power. If a true effect was weaker than pre-specified in the power analysis, fewer of the experiments would detect it than given by your power.
However, the power of the tests would still be at the nominal rate for effects of the size their power calculation was based on.
Concerning the further parameters to established: all of these of course depend on all other assumptions holding. These include


*

*normality

*equal variance (e.g. for a two-sample test)

*no multiple tests


If you violate one of these, the corrections will typically result in a change in $\alpha$ (if H0 is true) or $\beta$ (if H0 is false, since raising your significance threshold to e.g. control for multiplicity in effect reduces power).
