Does the Prospective-Alpha-Allocation-Sequence (PAAS) actually control the family-wise error rate?

Question

I recently read about the PAAS as an alternative to Bonferroni. Unlike Bonferroni, where $\alpha$ is subdivided among multiple tests such that

$$ \alpha_1 + \alpha_2 + \ldots + \alpha_k = \alpha $$

where $\alpha$ is the overall alpha. It is shown that Bonferroni controls the FWER (Family-wise Error Rate). I understood this to mean the probability that any test falsely rejects the null when the null is true, that is literally the interpretation of $\alpha$ for a well controlled test. It is optimal when the tests are independent.

Alternately, there is the PAAS which constrains the true negative rate:

$$ (1-\alpha_1)(1-\alpha_2)\ldots(1-\alpha_k) = (1-\alpha)$$

Immediately (for $k=2$) you can see PAAS "spends" the alpha less conservatively, so that instead of 0.025 as a cutoff for an evenly-split alpha $1-|\sqrt{1-0.05}| = 0.0253\ldots$. Performing the same calculation gives me the familywise error rate as $\text{Pr}(\text{Test 1 false rejects}) + \text{Pr}(\text{Test 2 false rejects}) = 0.0506\ldots$.

Is the FWER defined differently than this? Is PAAS an FDR method then?

Answer 1

Performing the same calculation gives me the familywise error rate as $\text{Pr}(\text{Test 1 false rejects}) + \text{Pr}(\text{Test 1 false rejects}) = 0.0506\ldots$

The probability of 1 or more tests failing is not the sum of the probabilities if the events may potentially overlap.

$$P(\text{A or B}) = P(\text{A}) + P(\text{B}) {\color{red}{ - P(\text{A and B})}}$$

The third term should make that you get a $0.05$ as result. That is, if the events are independent, such that $P(\text{A and B}) = P(\text{A}) \times P(\text{B})$. With the Bonferroni correction the worst case of $P(\text{A and B}) = 0$ is assumed.

Multiply, add, or condition on probability?

Answer 2

Moye published this approach in 1998 in annals of epidemiology and specially considers the case of reporting the results of a clinical trial so that a non-significant primary endpoint analysis would not render subsequent evaluation of secondary endpoints, but if the secondary endpoints are assessed, all such analyses should be presented. Moye argues that this is intuitive because most readership has an intuitive notion of the same alpha threshold for significance being assigned equally to equal tests, rather than in a step-down procedures where "lesser significant" results are tested at subsequently less stringent thresholds. A consequence of this is that the "overall alpha" of a trial is actually subjective and should be left to the reader to infer from the analyses presented. Specifically the below is written:

https://www.sciencedirect.com/science/article/pii/S1047279798000039?via%3Dihub

the following is a guide to investigators on the apportionment of alpha during the design phase of a research program. Commonly, the scientific community considers an alpha level for each hypothesis test to be examined in a research program, leaving the interpretation of the overall alpha to the reader who is attempting to interpret the significance of the study.

O'Neill and D'Agostino provided commentary with mixed feelings about the notion of lack of control of the overall study $\alpha$ in this scenario. Specifically D'Agostino nicely summarizes:

Doctor O'Neill's approach is different. He first praises Moye for his recognition of the problem and his contribution to addressing it, but then points to problems. These include the artificial classification of variables as primary and secondary, the problem of interpreting study results in light of a primary variable alpha and a new overall experiment alpha, and the overemphasis in the PAAS scheme of declaring each variable positive or negative in terms of alpha rather than examining the relation among the variables and the power of the study. He also makes useful suggestions to improve upon PAAS