After discovering alpha error inflation, how to estimate severity of the effect?

Question

Example: I find alpha inflation in an already published report. 300 indicators had been independently tested for statistical significance, and the "significant" results then reported as 95% confident without adjusted p-values. Multiple findings have already been taken as a fact, and decisions have been made based on that. What is the best way to quantify the severity of the problem, and make the severity of the effect intuitively understandable?.

I am aware of Sidak, Bonferonni, and False discovery rate, but they appear to be, at least primarily, ways to readjust the threshold for accepting the null hypothesis. In this context type II error cost is important, so simply adjusting the p-value doesn't fix the issue. The question is with regard to the general methodology of "blindly" measuring and testing as much as possible. How can the total risk of both errors be quantified in an intuitively understandable metric, in comparison for example to a more focused study with strong hypotheses.

Answer 1

It's possible there may be little problem at all; the 5% type I error rate means that of the nulls that were true 5% of those would be incorrectly rejected, but it's quite possible that none of the nulls were true. If considerably more than 5% of tests resulted in significance that would suggest that type I errors were not likely to be a major cause of rejections.

You'll likely want to at least mention that there's a risk that a number of the rejections may be type I errors, but aside from noting that it will be - at most - about 5% of the tests performed (and then only if there were no correct rejections at all), it may not require saying much more.

If you have access to the p-values, you could perhaps point out that the ones with p-values not substantially less than 5% would be one that would not have been rejected if the rate had been lowered to account for overall type I error.

However since decisions are being based on rejection or non-rejection, note that type II errors carry a cost as well - associated with failing to take whatever action would have been taken if the null had correctly been rejected. As such, lowering the overall type I error rate might in some situations be worse (more costly) in terms of the overall numbers and types of the two kinds of error.

Concern with familywise error rates perhaps makes some sense in a scientific context where type I errors loom large and someone may be trying to avoid much chance of making more than one across a large study with many tests; it may make less sense in (say) a business context to focus so hard on them.

Indeed, if the choice of 5% was based on a suitably selected tradeoff in the two error rates (I'd bet it wasn't, but if), then it would actually be better not to try to correct for familywise error because that would actually ruin the per-test tradeoff that was made.

In short: I'd advise a cautious response rather than a strong one. Don't focus only on overall type I error but consider also the effect on power (and its complement, type II error), and if possible the likely costs of the two errors (as well, if it's possible, the extent to which it's really plausible your nulls are actually exactly true -- one sided nulls may be, but exact truth of point nulls should be considered very carefully -- in many situations it's likely none are true).

For all you know at the moment, the best tradeoff between the two error types* across all of those tests would have been to raise, rather than lower the significance level.

*(if choosing a single significance level for all of them -- not necessarily the wisest course)