Skip to main content
deleted 6 characters in body
Source Link
Gala
  • 8.6k
  • 2
  • 32
  • 44

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities and can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest other approaches that sidestep the issue entirely.

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities and can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest other approaches that sidestep the issue entirely.

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities and can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest approaches that sidestep the issue entirely.

deleted 1 characters in body
Source Link
Gala
  • 8.6k
  • 2
  • 32
  • 44

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities thatand can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest other approaches that sidestep the issue entirely.

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities that can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest other approaches that sidestep the issue entirely.

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities and can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest other approaches that sidestep the issue entirely.

Source Link
Gala
  • 8.6k
  • 2
  • 32
  • 44

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities that can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest other approaches that sidestep the issue entirely.