Different types of error rates when there are multiple comparisons

Question

I have seen 5 different type of error rates. They include:

• Componentwise Error Rate
• Experimentwise Error Rate
• False Discovery Rate
• Strong Familywise Error Rate
• Simultaneous Confidence Intervals

Suppose $H_0 = H_{01} \cap \cdots \cap H_{0K}$ Here are some questions that I have:

1. So basically if the componentwise error rate is $\alpha$ then the probability of rejecting a single hypothesis in a single test is $\alpha$?
2. If the experimentwise error rate is $\alpha$ then the probability of rejecting any of the $H_{0i}$ when all of the $H_{0i}$ are true is $\alpha$?
3. What is the difference between the FDR and the experimentwise error rate? Doesn't the FDR control the experimentwise error rate?
4. Is the strong familywise error rate basically the strictest error rate? So the FDR allows for some slack if we have more correct rejections but the strong familywise error rate does not?
5. Simultaneous confidence intervals must cover their true parameter with probability $1-\alpha$. But a single confidence interval will cover its true parameter with probability greater than $1-\alpha$?

Answer 1

The terminology isn't standardized enough to be able to answer these questions definitively. Tentatively ...

(1) I haven't heard of 'componentwise', but would assume it means the same as 'comparisonwise'. If so, the answer is 'yes' when you append 'under the null hypothesis'.

(2) I think 'experimentwise' is a synonym of 'familywise'. So 'yes' again.

(3) FDR is a different approach. It doesn't try to control the chance of falsely rejecting a single true null (out of so many tested); rather it controls the expected proportion of true nulls rejected (out of so many tested).

(4) Strong control over the familywise error rate means that it's controlled under any subset of the null - meaning you can identify which individual $H_{0i}$ to reject. Weak control means it's controlled only under the full null - as in omnibus tests.

(5) For a bunch of simultaneous CIs, the chance that all of them cover their parameters is $(1-\alpha)$; the chance that each covers its parameter is greater. For a bunch of comparisonwise CIs the chance that each of them covers its parameter is $(1-\alpha)$; the chance that all cover their parameters is less.

But when you see one of these terms unaccompanied by a definition, be cautious in interpreting it.