# ANOVA with unreliable measure

I’m doing some literature review on sports science, in particular the effect of training on endurance.

There is no standard endurance tests. Some consist in intermittent contractions until exhaustion where the end force is measured.

Some tests have been the subject of studies to obtain ICCs, LoAs, or other inter-session reliability metrics, while some have not. Some have been found to have low ICC (< 0.8) or very large LoAs (+/- 40%).

I have come across some studies that look at the effect of training on endurance. They do an endurance test on a group, then subject them to training, and do another endurance test. They then do a one-way ANOVA to figure out whether there has been a significant increase in endurance. However, some of these studies use endurance tests that have not been characterized, there has never been any test-retest reliability evaluation.

So my question is : can an ANOVA conclude on significant differences if the measure used is unreliable, or has unknown reliability ?

It seems weird to me as this would be like using an unreliable scale without knowing its standard error.

(A brief answer, because there's lots that could be discussed here)

First, all measures are, to some degree, unreliable - it's very rare to find a perfect, noise-free measure of anything. Your issue is that some of the measures in question might be very unreliable.

Classical statistical tests like ANOVA can accomdate this unreliability. Briefly, ANOVA works by comparing the signal (the size of the differences in the group means) to the noise (the variability in scores within each group). With an unreliable measure, the variability within each group will be partly due to genuine differences between people, and partly due to the unreliability of the measure. You don't know the realiability, and you probably don't know the scale of the true differences, so you can't say how much of the noise is due to each factor.

ANOVA will still do what it's supposed to do: in situations where the null hypothesis is true, it will only indicate statistically significant results 5% of the time.

The issue here is that without any of this information, it's impossible to do a proper power analysis, and researchers end up either not doing any power analysis, or taking a wild guess at the likely effect size. This leads to all sorts of problems, discussed elsewhere.

• Thank you for your reply. Yes, you are right no measure is 100% reliable, but indeed my concern is that some of these tests have been found to be highly unreliable after the fact. I assume that if group differences are smaller (for some definition) than the LoAs, the ANOVA might find no statistically significant difference (?). In the problematic studies I have found, the estimate the effect size with eta^2, again without knowing the reliability of their measure. You said these problems are discussed elsewhere. Could you share some of these resources / links ? Commented Jul 20 at 10:31
• You might want to rewrite your second-to-last paragraph, which reads like a statement about the probability of the null hypothesis rather than the conditional probability of the data, given that the null hypothesis is true. Just a phrasing issue, but at the moment it doesn’t say what you intended it to. Commented Jul 21 at 8:07
• Classical statistical tests like ANOVA can accomdate this unreliability.---I don't think this is correct, at least in the sense that classical ANOVA designs work to estimate error. Because item-level measure is not included into the ANOVA decomposition, then there is no telling how much is due to the difference in means and how much is due to wildly inaccurate measures. This would of course be different if we were using ANOVA decomposition to explicitly identify sources of error in a measure, such as in generalizability theory. Commented Jul 21 at 15:11
• @ShawnHemelstrand, could you please clarify how what you're saying differs from "With an unreliable measure, the variability within each group will be partly due to genuine differences between people, and partly due to the unreliability of the measure. You don't know the realiability, and you probably don't know the scale of the true differences, so you can't say how much of the noise is due to each factor.", because I'm not seeing your distinction.
– Eoin
Commented Jul 22 at 11:46
• The statement you quoted here is obviously true, but it also says something completely different from the statement I quoted above. The first statement seems to indicate that ANOVA can deal with the unreliable measures, but in this quote it says the contrary. So the message in your answer to me doesn't have a unified message. ANOVA doesn't explicitly "accommodate" unreliable measures. As your quote here indicates, it masks the problem, which is precisely what OP asked about. Commented Jul 22 at 15:02

For a long time, a common response to your question would have been the caution that unreliable measures cause attenuation of effects, leading one to underestimate the true effects that occur. Adjustments for reliability typically proceeded according to this assumption; they date back to Charles Spearman, c. 1904. Eric Loken and Andrew Gelman have fairly recently (2017) shown that this principle holds only under certain conditions and that the amount of the noise and the size of the sample both matter. Sometimes unreliable measures will cause one to overestimate the size of effects. Therefore adjusting for attenuation can be informative but does not tell the whole story.

• Kline's book on SEM does a great job of explaining how complicated this can get. When you estimate fairly complicated models, the attenuation can propagate across the model, which makes disentangling the bias difficult if it is not explicitly determined or identified. Commented Jul 21 at 15:14
• @ShawnHemelstrand Is the book you're referring to "Beyond significance testing: Reforming data analysis methods in behavioral research"? Commented Jul 22 at 7:56
• Principles and Practice of Structural Equation Modeling by Rex Kline. @Experience111 If you instead need a book purely on measurement and reliability, Measurement Theory and Applications for the Social Sciences is also good. Commented Jul 22 at 15:03
• Thank you @ShawnHemelstrand. Interesting that it says "for the social sciences" in the title when this seems pretty integral to many types of natural sciences Commented Jul 22 at 16:13

So my question is: can an ANOVA conclude on significant differences if the measure used is unreliable, or has unknown reliability?

And my simple answer is "probably not", but that will depend on a lot of factors. To keep a very simple example of how this can be a problem I give a basic case here. We will pretend we designed an aptitude test which is scored from 0 to 100. We first determine what the standard error of measurement (SEM) is for this measure, which is defined as:

$$\text{SEM} = \sigma \sqrt{1 - \rho}$$

where $$\sigma$$ is the standard deviation of a measure and $$\rho$$ is the estimated reliability. Suppose someone scores 70 out of 100 on our test, but the reliability is $$\rho = .80$$ and the standard deviation of the measure is 10. The SEM for the test would consequently be:

$$10 \sqrt{1 - .80} = 4.472$$

Placing an upper and lower bound of two SEMs around the measure, this means that the person's "true" score is around $$70 \pm 8.944$$, meaning that their population value is actually approximately somewhere between 61 and 79. Now this is a problem, because if one person can vary this dramatically on a test, then everyone can, which means that the group means can, the prime target in ANOVA. The group differences may be almost completely attributable to very noisy measures which do not truly estimate the population level difference of means between groups, and consequently cannot be relied upon. How much of an issue this truly is will depend of course on the reliability and SEM of the measure.

• Thank you for this reply. I'm confused as to how the SEM is calculated exactly though. The link you provided mentions the standard deviation of the test, but I'm not sure how this is determined. Same for the terms that allow for the calculation of reliability. How can I know the standard deviation of the test, or variance of the true score? The only test-retest reliability evaluation work I've seen was focused on the ICC of the conceived aptitude test. Commented Jul 22 at 7:54
• Suppose you have 10 scores on this test. Then the standard deviation is simply the standard deviation of the 10 scores. The calculation above is for coefficient alpha, which is a measure of internal consistency. This is not the same as ICC, which in this context measures test-retest reliability. However, the formula for calculating the SEM is the same, where $\rho$ can be replaced with the ICC value. See this paper for details on calculation. Commented Jul 22 at 15:36