3 changes hypothesis pair

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence iI will use that as context.

h0: %conversion of A=B
ha: %conversion of A>B


$$H_0: \quad \text{percentage conversion of } A \leq \text{percentage conversion of } B\\ H_1: \quad \text{percentage conversion of } A > \text{percentage conversion of } B$$

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a chi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is a classic case oh multiple testing and is wrong without some form of alpha correction like Bonferroni

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does that also constitute as a repeated comparison, even though the data and hypothesis are the same?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data.

Appreciate the help.

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence i will use that as context.

h0: %conversion of A=B
ha: %conversion of A>B

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a chi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is a classic case oh multiple testing and is wrong without some form of alpha correction like Bonferroni

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does that also constitute as a repeated comparison, even though the data and hypothesis are the same?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data.

Appreciate the help.

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence I will use that as context.

$$H_0: \quad \text{percentage conversion of } A \leq \text{percentage conversion of } B\\ H_1: \quad \text{percentage conversion of } A > \text{percentage conversion of } B$$

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a chi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is a classic case oh multiple testing and is wrong without some form of alpha correction like Bonferroni

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does that also constitute as a repeated comparison, even though the data and hypothesis are the same?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data.

2 added 82 characters in body

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence i will use that as context.

h0: %conversion of A=B ha: %conversion of A>B

h0: %conversion of A=B
ha: %conversion of A>B

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a t-test or chi-square testchi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is what is referred to as multiple testinga classic case oh multiple testing and is wrong without correcting forsome form of alpha. correction like Bonferroni

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does itthat also constitute as a repeated comparison, even though the data and hypothesis are the same?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data.

Appreciate the help.

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence i will use that as context.

h0: %conversion of A=B ha: %conversion of A>B

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a t-test or chi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is what is referred to as multiple testing and is wrong without correcting for alpha.

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does it constitute as repeated comparison?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data.

Appreciate the help.

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence i will use that as context.

h0: %conversion of A=B
ha: %conversion of A>B

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a chi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is a classic case oh multiple testing and is wrong without some form of alpha correction like Bonferroni

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does that also constitute as a repeated comparison, even though the data and hypothesis are the same?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data.

Appreciate the help.

1

Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused.

My confusion started with A/B testing and hence i will use that as context.

h0: %conversion of A=B ha: %conversion of A>B

1. This is my understanding of multiple testing from an A/B standpoint. Running the test for a sample of say 1000 each (Test and Control), then performing a t-test or chi-square test to check for significance. If the results are not significant then continuing the test for another 1000 and repeating the process until significance is reached. This is what is referred to as multiple testing and is wrong without correcting for alpha.

2. But if i run the test on a sample of 100K each and measure if the difference in % conversion between the 2 groups are significantly different once then this is not considered multiple comparison

3. In (2) if i perform chi-sq test on the same 100K x 2 population 10 times does it constitute as repeated comparison?

4. Similar to (2) but instead of just measuring the difference for only %conversion, if I also measure if the difference in height, weight etc are also significant between test and control without alpha correction is that also considered as multiple testing? These are separate hypotheses run on the same data.

Appreciate the help.