Skip to main content
edited body
Source Link
Kodiologist
  • 20.6k
  • 2
  • 44
  • 78

It's possible there may be little problem at all; the 5% type I error rate Meansmeans that of the nulls that were true 5% of those would be incorrectly rejected, but it's quite possible that none of the nulls were true. If considerably more than 5% of tests resulted in significance that would suggest that type I errors were not likely to be a major cause of rejections.

You'll likely want to at least mention that there's a risk that a number of the rejections may be type I errors, but aside from noting that it will be - at most - about 5% of the tests performed (and then only if there were no correct rejections at all), it may not require saying much more.

If you have access to the p-values, you could perhaps point out that the ones with p-values not substantially less than 5% would be one that would not have been rejected if the rate had been lowered to account for overall type I error.

However since decisions are being based on rejection or non-rejection, note that type II errors carry a cost as well - associated with failing to take whatever action would have been taken if the null had correctly been rejected. As such, lowering the overall type I error rate might in some situations be worse (more costly) in terms of the overall numbers and types of the two kinds of error.

Concern with familywise error rates perhaps makes some sense in a scientific context where type I errors loom large and someone may be trying to avoid much chance of making more than one across a large study with many tests; it may make less sense in (say) a business context to focus so hard on them.

Indeed, if the choice of 5% was based on a suitably selected tradeoff in the two error rates (I'd bet it wasn't, but if), then it would actually be better not to try to correct for familywise error because that would actually ruin the per-test tradeoff that was made.

In short: I'd advise a cautious response rather than a strong one. Don't focus only on overall type I error but consider also the effect on power (and its complement, type II error), and if possible the likely costs of the two errors (as well, if it's possible, the extent to which it's really plausible your nulls are actually exactly true -- one sided nulls may be, but exact truth of point nulls should be considered very carefully -- in many situations it's likely none are true).

For all you know at the moment, the best tradeoff between the two error types* across all of those tests would have been to raise, rather than lower the significance level.

*(if choosing a single significance level for all of them -- not necessarily the wisest course)

It's possible there may be little problem at all; the 5% type I error rate Means that of the nulls that were true 5% of those would be incorrectly rejected, but it's quite possible that none of the nulls were true. If considerably more than 5% of tests resulted in significance that would suggest that type I errors were not likely to be a major cause of rejections.

You'll likely want to at least mention that there's a risk that a number of the rejections may be type I errors, but aside from noting that it will be - at most - about 5% of the tests performed (and then only if there were no correct rejections at all), it may not require saying much more.

If you have access to the p-values, you could perhaps point out that the ones with p-values not substantially less than 5% would be one that would not have been rejected if the rate had been lowered to account for overall type I error.

However since decisions are being based on rejection or non-rejection, note that type II errors carry a cost as well - associated with failing to take whatever action would have been taken if the null had correctly been rejected. As such, lowering the overall type I error rate might in some situations be worse (more costly) in terms of the overall numbers and types of the two kinds of error.

Concern with familywise error rates perhaps makes some sense in a scientific context where type I errors loom large and someone may be trying to avoid much chance of making more than one across a large study with many tests; it may make less sense in (say) a business context to focus so hard on them.

Indeed, if the choice of 5% was based on a suitably selected tradeoff in the two error rates (I'd bet it wasn't, but if), then it would actually be better not to try to correct for familywise error because that would actually ruin the per-test tradeoff that was made.

In short: I'd advise a cautious response rather than a strong one. Don't focus only on overall type I error but consider also the effect on power (and its complement, type II error), and if possible the likely costs of the two errors (as well, if it's possible, the extent to which it's really plausible your nulls are actually exactly true -- one sided nulls may be, but exact truth of point nulls should be considered very carefully -- in many situations it's likely none are true).

For all you know at the moment, the best tradeoff between the two error types* across all of those tests would have been to raise, rather than lower the significance level.

*(if choosing a single significance level for all of them -- not necessarily the wisest course)

It's possible there may be little problem at all; the 5% type I error rate means that of the nulls that were true 5% of those would be incorrectly rejected, but it's quite possible that none of the nulls were true. If considerably more than 5% of tests resulted in significance that would suggest that type I errors were not likely to be a major cause of rejections.

You'll likely want to at least mention that there's a risk that a number of the rejections may be type I errors, but aside from noting that it will be - at most - about 5% of the tests performed (and then only if there were no correct rejections at all), it may not require saying much more.

If you have access to the p-values, you could perhaps point out that the ones with p-values not substantially less than 5% would be one that would not have been rejected if the rate had been lowered to account for overall type I error.

However since decisions are being based on rejection or non-rejection, note that type II errors carry a cost as well - associated with failing to take whatever action would have been taken if the null had correctly been rejected. As such, lowering the overall type I error rate might in some situations be worse (more costly) in terms of the overall numbers and types of the two kinds of error.

Concern with familywise error rates perhaps makes some sense in a scientific context where type I errors loom large and someone may be trying to avoid much chance of making more than one across a large study with many tests; it may make less sense in (say) a business context to focus so hard on them.

Indeed, if the choice of 5% was based on a suitably selected tradeoff in the two error rates (I'd bet it wasn't, but if), then it would actually be better not to try to correct for familywise error because that would actually ruin the per-test tradeoff that was made.

In short: I'd advise a cautious response rather than a strong one. Don't focus only on overall type I error but consider also the effect on power (and its complement, type II error), and if possible the likely costs of the two errors (as well, if it's possible, the extent to which it's really plausible your nulls are actually exactly true -- one sided nulls may be, but exact truth of point nulls should be considered very carefully -- in many situations it's likely none are true).

For all you know at the moment, the best tradeoff between the two error types* across all of those tests would have been to raise, rather than lower the significance level.

*(if choosing a single significance level for all of them -- not necessarily the wisest course)

Source Link
Glen_b
  • 290.5k
  • 37
  • 652
  • 1.1k

It's possible there may be little problem at all; the 5% type I error rate Means that of the nulls that were true 5% of those would be incorrectly rejected, but it's quite possible that none of the nulls were true. If considerably more than 5% of tests resulted in significance that would suggest that type I errors were not likely to be a major cause of rejections.

You'll likely want to at least mention that there's a risk that a number of the rejections may be type I errors, but aside from noting that it will be - at most - about 5% of the tests performed (and then only if there were no correct rejections at all), it may not require saying much more.

If you have access to the p-values, you could perhaps point out that the ones with p-values not substantially less than 5% would be one that would not have been rejected if the rate had been lowered to account for overall type I error.

However since decisions are being based on rejection or non-rejection, note that type II errors carry a cost as well - associated with failing to take whatever action would have been taken if the null had correctly been rejected. As such, lowering the overall type I error rate might in some situations be worse (more costly) in terms of the overall numbers and types of the two kinds of error.

Concern with familywise error rates perhaps makes some sense in a scientific context where type I errors loom large and someone may be trying to avoid much chance of making more than one across a large study with many tests; it may make less sense in (say) a business context to focus so hard on them.

Indeed, if the choice of 5% was based on a suitably selected tradeoff in the two error rates (I'd bet it wasn't, but if), then it would actually be better not to try to correct for familywise error because that would actually ruin the per-test tradeoff that was made.

In short: I'd advise a cautious response rather than a strong one. Don't focus only on overall type I error but consider also the effect on power (and its complement, type II error), and if possible the likely costs of the two errors (as well, if it's possible, the extent to which it's really plausible your nulls are actually exactly true -- one sided nulls may be, but exact truth of point nulls should be considered very carefully -- in many situations it's likely none are true).

For all you know at the moment, the best tradeoff between the two error types* across all of those tests would have been to raise, rather than lower the significance level.

*(if choosing a single significance level for all of them -- not necessarily the wisest course)