4
$\begingroup$

I just started reading an article in Significance by Neil Sheldon (“What does it All Mean?,” Significance, 16:4, 15–17), which discusses the use and abuse of the term “significance” and the recent statements from the ASA in The American Statistician, and discussions thereof. As soon as the first paragraph, I read:

An old argument, one that can be found in R.A. Fisher’s early writings on the subject, makes the point that statistical significance is not the same as practical significance. In more recent years, the practical significance of a result has been quantified by the use of effect sizes. Very roughly, the effect size provides an indication of the practical significance of a finding.

I certainly agree with the first sentence. But the rest??? Is this widely accepted? Maybe I’m not giving enough import to “very roughly,” but from my own perspective, the only time an effect size measures practical significance is when it is zero.

My reasoning is that the way one should assess practical significance is by serious evaluation of the observed effect from the perspective of the underlying science. To do that, one needs to see the effect in the same units as the response scale. For example, how big a change in SBP, in mmHg, is attributable to that drug? Effect sizes, however, remove the response scale from consideration, and I’d argue that that is the antithesis of judging the true importance of the effect, and substitutes a relative change in standard-deviation units. So instead of quantifying practical significance, we are quantifying how big the effect is relative to how well we can measure it.

All this said, I do see that there are others who seem to support what Sheldon says. Within this forum, for example, I can refer you to https://stats.stackexchange.com/a/271268/52554 and Practical significance, especially with percents: "standard" measure and threshold, among others. Also, I can see an argument for standardizing an effect in another way, as a percentage change—e.g., we might be able to judge the practical significance of decreasing or increasing SBP by, say, 10 percent.

As a concrete illustration, consider a case where two experimenters conduct a study comparing two methods of producing rivets to be used in assembling airplane wings. It’s important the the diameters of the rivets be kept to a close tolerance. Suppose that both experimenters observe the same average difference of 0.2 mm between the two methods. But one experimenter’s measurements were taken using a wooden ruler, while the second used a micrometer. Thus the error SDs are vastly different, making the first experimenter’s effect size much smaller than the second’s. I argue that the practical importance of the result is the same in both experiments, and should be judged by whether the 0.2mm discrepancy is important.

In summary, I don’t think we are seriously addressing the distinction between statistical and practical significance by replacing one simplistic, highly abused practice with another one that is equally sloppy. Somewhere along the line, people should be thinking about the actual results achieved—-not looking for another way to avoid thinking carefully.

But, my question: When is it appropriate to measure practical significance using an effect-size measure, and why?

$\endgroup$
  • 3
    $\begingroup$ I think you answered your own question :P $\endgroup$ – Sheridan Grant Oct 22 at 19:33
  • $\begingroup$ Right: is there really a question here we need to answer? $\endgroup$ – whuber Oct 22 at 19:43
  • 1
    $\begingroup$ Well, there are question marks in there :-) But good point. I’ll edit so there is an actual question to be answered. $\endgroup$ – rvl Oct 22 at 20:00
  • 2
    $\begingroup$ I really wish “significant” never caught on as a term. From a semantics standpoint, it is too easy to map the concept on to ideas of “importance”, “meaningful”, and other misguided ideas that shape the term’s interpretation. Maybe it is the Bayesian in me, but I find “credible” to be a less loaded and more accurate summary of most “significant” effects. After all, something that is credible is scientifically interesting and can still end up being unsupported by subsequently acquired data. $\endgroup$ – Matt Barstead Oct 22 at 23:36
  • $\begingroup$ I agree “credible” is more appropriate. $\endgroup$ – rvl Oct 23 at 1:30
2
$\begingroup$

The answer to your question is implicit in your question, but can be made explicit: NEVER. Statistical summaries scrub study results of context and background and therefore no matter how useful they are as a component of (or a form of) reasoned argument, they should never be substituted for the thoughtful consideration of a human.

You are absolutely right in saying that practical significance is a matter for the science rather than the statistics. Yet another context-free rule of thumb for inference is the opposite of what is needed because scientists are already too often encouraged to substitute statistical recipes and rules for thoughtful consideration of the available information.

Statisticians should think hard before recommending yet another yardstick or threshold like P<0.05 or the 'small', 'medium' and 'large' effect size categories of Cohen. I have recently arXived a paper in which I dig deeply into the topic of the role of p-values in the scientific inferences of scientists (in the specific context of basic pharmacology) and I think it has relevance to this question. https://arxiv.org/abs/1910.02042

$\endgroup$
  • $\begingroup$ Well, I largely agree, obviously, but “never” is pretty strong. $\endgroup$ – rvl Oct 23 at 1:34
  • 1
    $\begingroup$ @rvl Perhaps there is a little hyperbole in my "NEVER", so you should feel free to read it as 'never'. $\endgroup$ – Michael Lew Oct 23 at 1:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.