6
$\begingroup$

I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size.

The problem is that I do not know of a "standard" way of doing so. Many people say that there is no standard way to measure practical significance -- it all depends on the problem. I completely agree, but I need to be able to cite someone authoritative to back up what I am doing. I think this is the reason that statistical significance testing is so prevalent -- there is a standard way of doing it: p < .05. So anyone can do it with little thinking. The way to get practical significance testing to be more popular might be to take a similar approach.

More specifically, I often need to see if two percents are practically different from each other. What is a good way of doing so -- something with an authoritative citation and which is easy to explain intuitively?

Odds ratio? What's a good citation for using it to measure practical significance? In health / social sciences would be a plus. Odds ratios are difficult to present intuitively though. Ideas on that?

Going one step further, what is a good "standard" / "magical" threshold for an OR (or whatever measure you like)? Like .05 for statistical significance testing. I know, I know, it depends on the problem. But is there a "standard" one with a good citation?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
6
$\begingroup$

I do a lot of statistical consulting and am commonly ask the question "How many subjects do I need to recruit?" I always address the sample size question in terms of "clinical" or "practical" significance. To address that I ask them to describe how large a difference "effect size" would they want to have to claim a difference between the groups. What you call a test of "statistical" significance is really a test of practical significance because even though the critical value is set based on a standard statistical significance level such as 0.05, the sample size is dictated by the power to detect a clinically or practically significant difference. There is no standar way to do this because the level that is called clinically significant is subjective and rightfully so.

Improve this answer
$\endgroup$
4
$\begingroup$

I don't think there is any kind of test for practical significance because it really depends on the field or the problem. Sorry.

For example, if you find that implementing a certain policy increases revenue by $0.0000001, (some negligibly small amount) even though this might be statistically significant, i.e, p-value < 0.05, practically the policy has negligible effect, so it is not practically significant. But the only way to tell about whether results are practically significant is too have good understanding of the problem. In some other cases coefficient of 0.0000001 could be very practically significant.

So basically, first you use statistical tools to see if something is statistically significant, and then use your knowledge of the field to see if something is practically significant. That's ideal case of course. Very often people try to pretend that their results are practically significant if they are statistically significant.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In the context of $y = a+bx$, I think a good measure of practical significance is $Q = b\sigma_x/\sigma_y$ (is there a name for this?). It measures the change (in SD's) in $y$ for a 1 SD change in $x$. Thus, you could say something like: Q < 1/3: very low effect; Q < 1/2: low effect; Q 1/2 to 2: medium effect; Q > 2: high effect; Q > 3: very high effect. In your example, all I need to know is that $b = 0.0000001$ and the SD's of $y$ and $x$. $\endgroup$
    – Jessica
    Jun 9, 2012 at 5:19
  • 1
    $\begingroup$ @Jessica Though I like your measure, I kind of agree with Akavall. A finding with a Q equal to 2 in personal psychology (where it is very difficult to find high correlations) is going to be viewed as more significant than if you find that kind of correlation in a gene expression study (where it is often easy to find correlations between members of the same signaling pathway). Not saying it can't be done, but I think you might want to look into some way of comparing effects to other known ones in the field. $\endgroup$
    – Andy McKenzie
    Jun 9, 2012 at 5:32
  • $\begingroup$ If you standardize your $x$-values by subtracting the mean & dividing by the SD, & standardize $Y$ similarly, you get the measure I think you're going for here. Namely, the slope becomes $r$, Pearson's correlation coefficient. $\endgroup$
    – gung - Reinstate Monica
    Jun 10, 2012 at 21:26
4
$\begingroup$

More than likely, if you're writing a paper, you're well on your way to what you need for practical significance. You've reviewed the literature and studied the subject matter and people have said, either explicitly or implicitly, what a practically significant amount is. All you need to do is cite that literature and use it in your article. If you want something just on the utility of effect sizes just google search effect size and your field (and perhaps cohen, 1962). Often there are effect size promotion papers specific to various disciplines. You could also look at Cohen 1962 as an example of how this kind of problem is approached (but not an example of what effect size is practically significant in your case-- it's unfortunate typical use).

There is no odds ratio that's a gold standard but I'm a bit surprised you're having a hard to explaining one. I suppose it's not surprising giving most textbook treatments. Betting uses odds ratios all of the time and people are familiar with it. If you need to explain it use that analogy. The odds of "Come by Chance" winning the race is 3:1 (or 3). Most people know what that means in payout. And they can see it's the number of times the horse is expected to lose the race (3) against the number of times it would win (1), or 75% of the time.

Of course, if this isn't just some proportion but something like a diagnostic odds ratio it's probably best to also be expressing things in specificity and sensitivity as well. Odds ratios alone miss bias in that case (and other similar ones where it is important).

Improve this answer
$\endgroup$
2
$\begingroup$

After further reading, here is one possible answer to my own question. If you have other answers along the same lines, please post them as well.

Cohen's h from Cohen (1988). $h = |2\arcsin\sqrt{p_1}-2\arcsin\sqrt{p_2}|$

Qualification of effect sizes, with the disclaimer that they might be different in other disciplines,

  • $h = 0.2$: "small"
  • $h = 0.5$: "medium"
  • $h = 0.8$: "large"

In R, http://rss.acs.unt.edu/Rdoc/library/pwr/html/ES.h.html

Improve this answer
$\endgroup$
1
  • $\begingroup$ effect-size is a very broad class of statistical estimation. It's not a bad potential answer to your question though. $\endgroup$
    – Nick Stauner
    Jan 10, 2014 at 1:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.