10
$\begingroup$

I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size.

The problem is that I do not know of a "standard" way of doing so. Many people say that there is no standard way to measure practical significance -- it all depends on the problem. I completely agree, but I need to be able to cite someone authoritative to back up what I am doing. I think this is the reason that statistical significance testing is so prevalent -- there is a standard way of doing it: p < .05. So anyone can do it with little thinking. The way to get practical significance testing to be more popular might be to take a similar approach.

More specifically, I often need to see if two percents are practically different from each other. What is a good way of doing so -- something with an authoritative citation and which is easy to explain intuitively?

Odds ratio? What's a good citation for using it to measure practical significance? In health / social sciences would be a plus. Odds ratios are difficult to present intuitively though. Ideas on that?

Going one step further, what is a good "standard" / "magical" threshold for an OR (or whatever measure you like)? Like .05 for statistical significance testing. I know, I know, it depends on the problem. But is there a "standard" one with a good citation?

$\endgroup$
1
  • $\begingroup$ practical significance is a vague term. please indicate its relevance to statistical significance. $\endgroup$ Commented Aug 24, 2023 at 13:33

5 Answers 5

11
$\begingroup$

I do a lot of statistical consulting and am commonly ask the question "How many subjects do I need to recruit?" I always address the sample size question in terms of "clinical" or "practical" significance. To address that I ask them to describe how large a difference "effect size" would they want to have to claim a difference between the groups. What you call a test of "statistical" significance is really a test of practical significance because even though the critical value is set based on a standard statistical significance level such as 0.05, the sample size is dictated by the power to detect a clinically or practically significant difference. There is no standar way to do this because the level that is called clinically significant is subjective and rightfully so.

$\endgroup$
2
  • $\begingroup$ What you call a test of "statistical" significance is really a test of practical significance because even though the critical value is set based on a standard statistical significance level such as 0.05, the sample size is dictated by the power to detect a clinically or practically. " - quite confusing. Validation and statistical methods do not have same genesis? How the two :statistical significance and practical or clinical significance can be equated? $\endgroup$ Commented Aug 26, 2023 at 4:23
  • 1
    $\begingroup$ @SubhashC.Davar Sadly Michael R. Chernick passed away 2 years ago. You can read obituaries here: magazine.amstat.org/blog/2021/03/01/… and here fluehr.com/obituary-detail.php?obitid=4630 . Maybe you should ask a new separate question; someone else might be able to offer you the information or clarification you're looking for in a detailed answer. $\endgroup$
    – J-J-J
    Commented Aug 28, 2023 at 6:44
8
$\begingroup$

I don't think there is any kind of test for practical significance because it really depends on the field or the problem. Sorry.

For example, if you find that implementing a certain policy increases revenue by $0.0000001, (some negligibly small amount) even though this might be statistically significant, i.e, p-value < 0.05, practically the policy has negligible effect, so it is not practically significant. But the only way to tell about whether results are practically significant is too have good understanding of the problem. In some other cases coefficient of 0.0000001 could be very practically significant.

So basically, first you use statistical tools to see if something is statistically significant, and then use your knowledge of the field to see if something is practically significant. That's ideal case of course. Very often people try to pretend that their results are practically significant if they are statistically significant.

$\endgroup$
3
  • $\begingroup$ In the context of $y = a+bx$, I think a good measure of practical significance is $Q = b\sigma_x/\sigma_y$ (is there a name for this?). It measures the change (in SD's) in $y$ for a 1 SD change in $x$. Thus, you could say something like: Q < 1/3: very low effect; Q < 1/2: low effect; Q 1/2 to 2: medium effect; Q > 2: high effect; Q > 3: very high effect. In your example, all I need to know is that $b = 0.0000001$ and the SD's of $y$ and $x$. $\endgroup$
    – Jessica
    Commented Jun 9, 2012 at 5:19
  • 1
    $\begingroup$ @Jessica Though I like your measure, I kind of agree with Akavall. A finding with a Q equal to 2 in personal psychology (where it is very difficult to find high correlations) is going to be viewed as more significant than if you find that kind of correlation in a gene expression study (where it is often easy to find correlations between members of the same signaling pathway). Not saying it can't be done, but I think you might want to look into some way of comparing effects to other known ones in the field. $\endgroup$ Commented Jun 9, 2012 at 5:32
  • $\begingroup$ If you standardize your $x$-values by subtracting the mean & dividing by the SD, & standardize $Y$ similarly, you get the measure I think you're going for here. Namely, the slope becomes $r$, Pearson's correlation coefficient. $\endgroup$ Commented Jun 10, 2012 at 21:26
6
$\begingroup$

More than likely, if you're writing a paper, you're well on your way to what you need for practical significance. You've reviewed the literature and studied the subject matter and people have said, either explicitly or implicitly, what a practically significant amount is. All you need to do is cite that literature and use it in your article. If you want something just on the utility of effect sizes just google search effect size and your field (and perhaps cohen, 1962). Often there are effect size promotion papers specific to various disciplines. You could also look at Cohen 1962 as an example of how this kind of problem is approached (but not an example of what effect size is practically significant in your case-- it's unfortunate typical use).

There is no odds ratio that's a gold standard but I'm a bit surprised you're having a hard to explaining one. I suppose it's not surprising giving most textbook treatments. Betting uses odds ratios all of the time and people are familiar with it. If you need to explain it use that analogy. The odds of "Come by Chance" winning the race is 3:1 (or 3). Most people know what that means in payout. And they can see it's the number of times the horse is expected to lose the race (3) against the number of times it would win (1), or 75% of the time.

Of course, if this isn't just some proportion but something like a diagnostic odds ratio it's probably best to also be expressing things in specificity and sensitivity as well. Odds ratios alone miss bias in that case (and other similar ones where it is important).

$\endgroup$
5
$\begingroup$

To add to most other answers, as you were looking for references, below you'll find some saying that you should avoid using arbitrary thresholds.

Note that there are various standardized effect sizes established by Jacob Cohen on a "small/medium/large" scale; you can find some specific examples on the wikipedia article about effect size. These thresholds or standards seem to be quoted in the documentation of some statistical software too. But if you dig a bit, you'll realize that even Cohen advised against using this "small/medium/large" standard -or any arbitrary standard, for that matter. See several of his quotes at the end of this answer, taken from the book where he established this "small/medium/large" standard.

Here are some reliable references about the problems caused by using standardized effect sizes:

Glass, G.V., B. McGaw, and M.L. Smith, in Meta-Analysis in Social Research (1981):

There is no wisdom whatsoever in attempting to associate regions of the effect size metric with descriptive adjectives such as “small,” “moderate,” “large,” and the like. Dissociated from a context of decision and comparative value, there is little inherent value to an effect size of 3.5 or .2. Depending on what benefits can be achieved at what cost, an effect size of 2.0 might be “poor” and one of .1 might be “good.”

Kelley, K. and Preacher, K.J., On Effect Size (2012), https://doi.org/10.1037/a0028086:

Consequently, as tempting as it may be, the idea of linking universal descriptive terms (e.g., “small,” “moderate,” or “large”) to specific effect sizes is largely unnecessary and at times misleading (e.g., Baguley, 2009; Lenth, 2001; Robinson et al., 2003; B. Thompson, 2002).

Our view is that the meaningfulness of an effect is inextricably tied to the particular area, research design, population of interest, and research goal, and it would be inappropriate to wed effect size to some necessarily arbitrary suggestion of substantive significance

Harrell, F., Statistical Problems to Document and to Avoid: Checklist for Authors (2020):

Many researchers use Cohen’s standardized effect sizes in planning a study. This has the advantage of not requiring pilot data. But such effect sizes are not biologically meaningful and may hide important issues as discussed by Lenth. Studies should be designed on the basis of effects that are relevant to the investigator and human subjects. [...]

Even Jacob Cohen, who established the "small/medium/large" thresholds in his book Statistical Power Analysis for the Behavioral Sciences (1988, second edition), repeatedly and unequivocally warned against their use, and considered them as a last resort. In other words, even if you consider they might have some value (which may be debated, as seen in previous quotes), Cohen considered that you should avoid them as much as possible. For example, page 25 of his book, about the effect size $d$:

The terms "small," "medium," and "large" are relative, not only to each other, but to the area of behavioral science or even more particularly to the specific content and research method being employed in any given investigation (see Sections 1.4 and 11.1). In the face of this relativity, there is a certain risk inherent in offering conventional operational definitions for these terms for use in power analysis in as diverse a field of inquiry as behavioral science. This risk is nevertheless accepted in the belief that more is to be gained than lost by supplying a common conventional frame of reference which is recommended for use only when no better basis for estimating the ES index is available.

p.113, about the effect size $q$:

Again, the reader is urged to avoid the use of these conventions, if he can, in favor of exact values provided by theory.

p.147, about the effect size $g$:

It must be reiterated, however, that a basis for positing $g$ which comes from theory or experience should automatically take precedence over these conventions.

p.184, about Cohen's $h$:

As before, the reader is counseled to avoid the use of these conventions, if he can, in favor of exact values provided by theory or experience in the specific area in which he is working.

p.224, about the effect size $\omega$:

The best guide here, as always, is the development of some sense of magnitude ad hoc, for a particular problem or a particular field.

and so on.

$\endgroup$
3
$\begingroup$

After further reading, here is one possible answer to my own question. If you have other answers along the same lines, please post them as well.

Cohen's h from Cohen (1988). $h = |2\arcsin\sqrt{p_1}-2\arcsin\sqrt{p_2}|$

Qualification of effect sizes, with the disclaimer that they might be different in other disciplines,

  • $h = 0.2$: "small"
  • $h = 0.5$: "medium"
  • $h = 0.8$: "large"

In R, http://rss.acs.unt.edu/Rdoc/library/pwr/html/ES.h.html

$\endgroup$
1
  • 4
    $\begingroup$ effect-size is a very broad class of statistical estimation. It's not a bad potential answer to your question though. $\endgroup$ Commented Jan 10, 2014 at 1:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.