I just started reading an article in Significance by Neil Sheldon (“What does it All Mean?,” Significance, 16:4, 15–17), which discusses the use and abuse of the term “significance” and the recent statements from the ASA in The American Statistician, and discussions thereof. As soon as the first paragraph, I read:
An old argument, one that can be found in R.A. Fisher’s early writings on the subject, makes the point that statistical significance is not the same as practical significance. In more recent years, the practical significance of a result has been quantified by the use of effect sizes. Very roughly, the effect size provides an indication of the practical significance of a finding.
I certainly agree with the first sentence. But the rest??? Is this widely accepted? Maybe I’m not giving enough import to “very roughly,” but from my own perspective, the only time an effect size measures practical significance is when it is zero.
My reasoning is that the way one should assess practical significance is by serious evaluation of the observed effect from the perspective of the underlying science. To do that, one needs to see the effect in the same units as the response scale. For example, how big a change in SBP, in mmHg, is attributable to that drug? Effect sizes, however, remove the response scale from consideration, and I’d argue that that is the antithesis of judging the true importance of the effect, and substitutes a relative change in standard-deviation units. So instead of quantifying practical significance, we are quantifying how big the effect is relative to how well we can measure it.
All this said, I do see that there are others who seem to support what Sheldon says. Within this forum, for example, I can refer you to https://stats.stackexchange.com/a/271268/52554 and Practical significance, especially with percents: "standard" measure and threshold, among others. Also, I can see an argument for standardizing an effect in another way, as a percentage change—e.g., we might be able to judge the practical significance of decreasing or increasing SBP by, say, 10 percent.
As a concrete illustration, consider a case where two experimenters conduct a study comparing two methods of producing rivets to be used in assembling airplane wings. It’s important the the diameters of the rivets be kept to a close tolerance. Suppose that both experimenters observe the same average difference of 0.2 mm between the two methods. But one experimenter’s measurements were taken using a wooden ruler, while the second used a micrometer. Thus the error SDs are vastly different, making the first experimenter’s effect size much smaller than the second’s. I argue that the practical importance of the result is the same in both experiments, and should be judged by whether the 0.2mm discrepancy is important.
In summary, I don’t think we are seriously addressing the distinction between statistical and practical significance by replacing one simplistic, highly abused practice with another one that is equally sloppy. Somewhere along the line, people should be thinking about the actual results achieved—-not looking for another way to avoid thinking carefully.
But, my question: When is it appropriate to measure practical significance using an effect-size measure, and why?
