Practical significance, especially with percents: "standard" measure and threshold

Question

I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size.

The problem is that I do not know of a "standard" way of doing so. Many people say that there is no standard way to measure practical significance -- it all depends on the problem. I completely agree, but I need to be able to cite someone authoritative to back up what I am doing. I think this is the reason that statistical significance testing is so prevalent -- there is a standard way of doing it: p < .05. So anyone can do it with little thinking. The way to get practical significance testing to be more popular might be to take a similar approach.

More specifically, I often need to see if two percents are practically different from each other. What is a good way of doing so -- something with an authoritative citation and which is easy to explain intuitively?

Odds ratio? What's a good citation for using it to measure practical significance? In health / social sciences would be a plus. Odds ratios are difficult to present intuitively though. Ideas on that?

Going one step further, what is a good "standard" / "magical" threshold for an OR (or whatever measure you like)? Like .05 for statistical significance testing. I know, I know, it depends on the problem. But is there a "standard" one with a good citation?

Answer 1

I do a lot of statistical consulting and am commonly ask the question "How many subjects do I need to recruit?" I always address the sample size question in terms of "clinical" or "practical" significance. To address that I ask them to describe how large a difference "effect size" would they want to have to claim a difference between the groups. What you call a test of "statistical" significance is really a test of practical significance because even though the critical value is set based on a standard statistical significance level such as 0.05, the sample size is dictated by the power to detect a clinically or practically significant difference. There is no standar way to do this because the level that is called clinically significant is subjective and rightfully so.

Answer 2

I don't think there is any kind of test for practical significance because it really depends on the field or the problem. Sorry.

For example, if you find that implementing a certain policy increases revenue by $0.0000001, (some negligibly small amount) even though this might be statistically significant, i.e, p-value < 0.05, practically the policy has negligible effect, so it is not practically significant. But the only way to tell about whether results are practically significant is too have good understanding of the problem. In some other cases coefficient of 0.0000001 could be very practically significant.

So basically, first you use statistical tools to see if something is statistically significant, and then use your knowledge of the field to see if something is practically significant. That's ideal case of course. Very often people try to pretend that their results are practically significant if they are statistically significant.

Answer 3

More than likely, if you're writing a paper, you're well on your way to what you need for practical significance. You've reviewed the literature and studied the subject matter and people have said, either explicitly or implicitly, what a practically significant amount is. All you need to do is cite that literature and use it in your article. If you want something just on the utility of effect sizes just google search effect size and your field (and perhaps cohen, 1962). Often there are effect size promotion papers specific to various disciplines. You could also look at Cohen 1962 as an example of how this kind of problem is approached (but not an example of what effect size is practically significant in your case-- it's unfortunate typical use).

There is no odds ratio that's a gold standard but I'm a bit surprised you're having a hard to explaining one. I suppose it's not surprising giving most textbook treatments. Betting uses odds ratios all of the time and people are familiar with it. If you need to explain it use that analogy. The odds of "Come by Chance" winning the race is 3:1 (or 3). Most people know what that means in payout. And they can see it's the number of times the horse is expected to lose the race (3) against the number of times it would win (1), or 75% of the time.

Of course, if this isn't just some proportion but something like a diagnostic odds ratio it's probably best to also be expressing things in specificity and sensitivity as well. Odds ratios alone miss bias in that case (and other similar ones where it is important).

Answer 4

After further reading, here is one possible answer to my own question. If you have other answers along the same lines, please post them as well.

Cohen's h from Cohen (1988). $h = |2\arcsin\sqrt{p_1}-2\arcsin\sqrt{p_2}|$

Qualification of effect sizes, with the disclaimer that they might be different in other disciplines,

• $h = 0.2$: "small"
• $h = 0.5$: "medium"
• $h = 0.8$: "large"

In R, http://rss.acs.unt.edu/Rdoc/library/pwr/html/ES.h.html

Answer 5

To add to most other answers, as you were looking for references, below you'll find some saying that you should avoid using arbitrary thresholds.

Note that there are various standardized effect sizes established by Jacob Cohen on a "small/medium/large" scale; you can find some specific examples on the wikipedia article about effect size. These thresholds or standards seem to be quoted in the documentation of some statistical software too. But if you dig a bit, you'll realize that even Cohen advised against using this "small/medium/large" standard -or any arbitrary standard, for that matter. See several of his quotes at the end of this answer, taken from the book where he established this "small/medium/large" standard.

Here are some reliable references about the problems caused by using standardized effect sizes:

Glass, G.V., B. McGaw, and M.L. Smith, in Meta-Analysis in Social Research (1981):

There is no wisdom whatsoever in attempting to associate regions of the effect size metric with descriptive adjectives such as “small,” “moderate,” “large,” and the like. Dissociated from a context of decision and comparative value, there is little inherent value to an effect size of 3.5 or .2. Depending on what benefits can be achieved at what cost, an effect size of 2.0 might be “poor” and one of .1 might be “good.”

Kelley, K. and Preacher, K.J., On Effect Size (2012), https://doi.org/10.1037/a0028086:

Consequently, as tempting as it may be, the idea of linking universal descriptive terms (e.g., “small,” “moderate,” or “large”) to specific effect sizes is largely unnecessary and at times misleading (e.g., Baguley, 2009; Lenth, 2001; Robinson et al., 2003; B. Thompson, 2002).

Our view is that the meaningfulness of an effect is inextricably tied to the particular area, research design, population of interest, and research goal, and it would be inappropriate to wed effect size to some necessarily arbitrary suggestion of substantive significance

Harrell, F., Statistical Problems to Document and to Avoid: Checklist for Authors (2020):

Many researchers use Cohen’s standardized effect sizes in planning a study. This has the advantage of not requiring pilot data. But such effect sizes are not biologically meaningful and may hide important issues as discussed by Lenth. Studies should be designed on the basis of effects that are relevant to the investigator and human subjects. [...]

Even Jacob Cohen, who established the "small/medium/large" thresholds in his book Statistical Power Analysis for the Behavioral Sciences (1988, second edition), repeatedly and unequivocally warned against their use, and considered them as a last resort. In other words, even if you consider they might have some value (which may be debated, as seen in previous quotes), Cohen considered that you should avoid them as much as possible. For example, page 25 of his book, about the effect size $d$:

The terms "small," "medium," and "large" are relative, not only to each other, but to the area of behavioral science or even more particularly to the specific content and research method being employed in any given investigation (see Sections 1.4 and 11.1). In the face of this relativity, there is a certain risk inherent in offering conventional operational definitions for these terms for use in power analysis in as diverse a field of inquiry as behavioral science. This risk is nevertheless accepted in the belief that more is to be gained than lost by supplying a common conventional frame of reference which is recommended for use only when no better basis for estimating the ES index is available.

p.113, about the effect size $q$:

Again, the reader is urged to avoid the use of these conventions, if he can, in favor of exact values provided by theory.

p.147, about the effect size $g$:

It must be reiterated, however, that a basis for positing $g$ which comes from theory or experience should automatically take precedence over these conventions.

p.184, about Cohen's $h$:

As before, the reader is counseled to avoid the use of these conventions, if he can, in favor of exact values provided by theory or experience in the specific area in which he is working.

p.224, about the effect size $\omega$:

The best guide here, as always, is the development of some sense of magnitude ad hoc, for a particular problem or a particular field.

and so on.