I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence i will use that as context.

h0: %conversion of A=B
ha: %conversion of A>B

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a t-test or chi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is what is referred to as multiple testing and is wrong without correcting for alpha.

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does it constitute as repeated comparison?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data. 

Appreciate the help.