Dependent tests?

Question

I have 3 independent data groups: A, B, C. I performed t-test on each pair of groups: AB, AC and BC. So these three tests (AB, AC, BC) are not independent, are they? Thanks for answer!

Answer 1

That's right. The three tests are not independent because they have data in common. That means that you have to consider two issues when interpreting the outcomes of these three tests -- alpha inflation and the lack of independence.

The following discussion is from pp. 265-271 of Sauro, J., & Lewis, J. R. (2016). Quantifying the User Experience: Practical Statistics for User Research, 2nd Ed. Cambridge, MA: Morgan-Kaufmann (you can look inside this book at https://www.amazon.com/Quantifying-User-Experience-Second-Statistics/dp/0128023082/).

---

WHAT IF YOU NEED TO RUN MORE THAN ONE TEST?

What if you have collected data from three groups instead of two, and want to compare Group A with B, A with C, and B with C? You’ll need to perform multiple comparisons. As Cowles (1989, p. 171) pointed out, this has been a controversial topic in statistics for decades. ...

ON ONE HAND

When the null hypothesis of no difference is true, you can think of a single test with α = 0.05 as the flip of a single coin that has a 95% chance of heads (correctly failing to reject the null hypothesis) and a 5% chance of tails (falsely concluding there is a difference when there really isn’t one—a false alarm, a Type I error). These are the probabilities for a single toss of the coin (a single test), but what if you run more than one test? Statisticians sometimes make a distinction between the error rate per comparison (EC) and the error rate per family (EF, or family-wise error rate) (Myers, 1979).

For example, if you ran 20 t-tests after collecting data in a usability study and there was really no difference in the tested products, you’d expect one Type I error, falsely concluding that there was a difference when that outcome happened just by chance. Unfortunately, other possible outcomes, such as seeing two or three Type I errors, also have a reasonable likelihood of happening by chance. The technical term for this is alpha inflation. For this series of tests, the actual value of α (defining α as the likelihood of getting one or more Type I errors) is much higher than 0.05. Table 9.4 shows, as expected, that the most likely number of Type I errors in a set of 20 independent tests with α = 0.05 is one, with a point probability of 0.37735. The likelihood of at least one Type I error, however, is higher—as shown in Table 9.4, it’s 0.64151. So, rather than having a 5% chance of encountering a Type I error when there is no real difference, α has inflated to about 64%. ...

Since the middle of the 20th century, there have been many strategies and techniques published to guide the analysis of multiple comparisons (Abelson, 1995; Cliff, 1987; Myers, 1979; Winer et al., 1991), such as omnibus tests (e.g., ANOVA and MANOVA) and procedures for the comparison of pairs of means (e.g., Tukey’s WSD and HSD procedures, the Student–Newman–Keuls test, Dunnett’s test, the Duncan procedure, the Scheffé procedure, the Bonferroni adjustment, and the Benjamini–Hochberg adjustment). A detailed discussion of all these techniques for reducing the effect of alpha inflation on statistical decision-making is beyond the scope of this book.

A popular and conceptually simple approach to controlling alpha inflation is the Bonferroni adjustment (Cliff, 1987; Myers, 1979; Winer et al., 1991). To apply the Bonferroni adjustment, divide the desired overall level of alpha by the number of tests you plan to run. For example, to run 10 tests for a family-wise error rate of 0.05, you would set α = 0.005 for each individual test. For 20 tests, it would be 0.0025 (0.05/20). Setting α = 0.0025 and running 20 independent tests would result in alpha inflation bringing the family-wise error rate to just under 0.05:

p(at least one Type I error) = 1−(1−0.0025)^20 = 1−0.9975^20 = 1−0.9512 = 0.0488

A relatively new method called the Benjamini–Hochberg adjustment (Benjamini & Hochberg, 1995) offers a good balance between making the Bonferroni adjustment and no adjustment at all. Rather than using a significance threshold of α/k (where k is the number of comparisons) for all multiple comparisons (the Bonferroni approach), the Benjamini–Hochberg method produces a graduated series of significance thresholds. To use the method, take the p-values from all the comparisons and rank them from lowest to highest. Then create a new threshold for statistical significance by dividing the rank by the number of comparisons and then multiplying this by the initial significance threshold (alpha). The first threshold will always be the same as the Bonferroni threshold, and the last threshold will always be the same as the unadjusted value of α. The thresholds in between the first and last comparisons rise in equal steps from the Bonferroni to the unadjusted threshold.

Problem solved—or is it?

ON THE OTHER HAND

When the null hypothesis is not true, applying techniques such as Bonferroni or Benjamini–Hochberg adjustments can increase the number of Type II errors—the failure to detect differences that are really there (misses as opposed to the false alarms of Type I errors) (Abelson, 1995; Myers, 1979; Perneger, 1998; Winer et al., 1991). As illustrated in Table 9.2, an overemphasis on the prevention of Type I errors leads to the proliferation of Type II errors. Unless, for your situation, the cost of a Type I error is much greater than the cost of a Type II error, you should avoid applying any of the techniques designed to suppress alpha inflation, including Bonferroni or Benjamini–Hochberg adjustments. “Simply describing what tests of significance have been performed, and why, is generally the best way of dealing with multiple comparisons” (Perneger, 1998, p. 1236).

OUR RECOMMENDATION

Abelson (1995, p. 70) stated: "When there are multiple tests within the same study or series of studies, a stylistic issue is unavoidable. As Diaconis (1985) put it, 'Multiplicity is one of the most prominent difficulties with data analytic procedures. Roughly speaking, if enough different statistics are computed, some of them will be sure to show structure' (p. 9). In other words, random patterns will seem to contain something systematic when scrutinized in many particular ways. If you look at enough boulders, there is bound to be one that looks like a sculpted human face. Knowing this, if you apply extremely strict criteria for what is to be recognized as an intentionally carved face, you might miss the whole show on Easter Island."

As discussed throughout this chapter, user researchers need to balance confidence and power in their studies, avoiding excessive attention to Type I errors over Type II errors unless the relative cost of a Type I error (thinking an effect is real when it isn’t) is much greater than that of a Type II error (failing to find and act upon real effects). This general strategy applies to the treatment of multiple comparisons just as it did in the previous discussions of one- versus two-tailed testing and the legitimacy of setting α > 0.05.

For most situations, we encourage user researchers to follow Perneger’s (1998) advice to run multiple comparisons at the designated level of alpha, making sure to report what tests have been done and why. For example, in summative usability tests, most practitioners use a fairly small set of well-defined and conventional measurements (success rates, completion times, user satisfaction) collected in a carefully constructed set of test scenarios, either for purposes of estimation or comparison with benchmarks or a fairly small and carefully selected set of products/systems. This practice helps to legitimize multiple testing at a specified and not overly conservative level of alpha because the researchers have clear a priori hypotheses under test. ...

One quick note—these computations assume that the multiple tests are independent, which will rarely be the case, especially when conducting within-subjects studies or doing multiple comparisons of all pairs of products in a multiproduct study. Fortunately, according to Winer et al. (1991), dependence among tests reduces the extent of alpha inflation as a function of the degree of dependence. This means that acting as if the data are independent even when they are not is consistent with a relatively conservative approach to this aspect of data analysis. The potential complexities of accounting for dependencies among data are beyond the scope of this book, and are not necessary for most practical user research.

If there is a need to adjust thresholds of significance for multiple comparisons, we recommend the Benjamini–Hochberg procedure due to its placement between liberal (unadjusted) and conservative (Bonferroni) approaches, but keep in mind that there is no single correct method for all situations.

References

Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum.

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Society, Series B (Methodological), 57(1), 289-300.

Cliff, N. (1987). Analyzing multivariate data. San Diego, CA: Harcourt, Brace, Jovanovich.

Cowles, M. (1989). Statistics in psychology: An historical perspective. Hillsdale, NJ: Lawrence Erlbaum.

Myers, J. L. (1979). Fundamentals of experimental design, 3rd ed. Boston, MA: Allyn and Bacon.

Perneger, T. V. (1998). What’s wrong with Bonferroni adjustments? British Medical Journal, 316, 1236-1238.

Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design, 3rd ed. New York, NY: McGraw-Hill.

Answer 2

Jim Lewis is correct that the tests described in the question are not independent. However, I see several problems with Lewis' advice on how to handle the problem of multiple comparisons.

For example, it is not correct that the error rate per family is the same as the familywise error rate (Frane, 2015). Moreover, the Benjamini-Hochberg procedure (which Lewis recommended) doesn't control either of those error rates--it controls a less stringent rate that wasn't mentioned, called the false discovery rate (Benjamini & Hochberg, 1995). False discovery rate control is typically reserved for large numbers of hypotheses--not for 3 tests as in the question. Also, like the other responses, Lewis' provides a general recommendation (in this case, either not adjusting at all or using the Benjamini-Hochberg procedure) without knowing the necessary information about the situation (e.g., in a clinical trial, neither of those recommended approaches would be acceptable).

Lewis also appears to endorse the do-nothing approach of Perneger, which has been heavily criticized (e.g., see here, here, and here). Lewis' explanation for this endorsement was: "Unless, for your situation, the cost of a Type I error is much greater than the cost of a Type II error, you should avoid applying any of the techniques designed to suppress alpha inflation, including Bonferroni or Benjamini–Hochberg adjustments." That doesn't seem to make sense. Type I errors are typically considered much more costly than Type II errors in null hypothesis testing--that's why we test at alpha = .05. If Type II errors were just as bad as Type I errors, wouldn't we be testing at alpha = .5? The whole purpose of null hypothesis testing in the first place is to prevent Type I errors.