1
$\begingroup$

I am unable to understand content reported in : Practical vs Statistical significance

Statistical significance-testing implies whether a sample-statistic matches with population estimate of effect-size.(mean,r or d etc.).It is based on sampling theory. An interpretation of the practical significance is vague and obscure. It could be that it reflects validity-coefficient as discussed in psychometric methods or something else.

My concern is how does it differ from the concept of "statistical" significance? And which of the two types of significance is better?

$\endgroup$
4

3 Answers 3

10
$\begingroup$

Explanation

The term "practical significance" is often an attempt to subdue the often erroneous usage of "significance" in statistics, which is supposed to be shorthand for "statistical significance." When an inferential test is "statistically significant," this is supposed to equate the probability of finding the result given from a test if the null hypothesis is true. If we surpass the alpha threshold usually given (in my field .05), we can reject the null hypothesis (that there is no effect). What this doesn't say at all is whether the null hypothesis is actually true or if the alternative hypothesis is consequently true either. There is a lot of literature about the problematic usage of null hypothesis significance testing (NHST), which you can read about here.

This is why sometimes you will hear people ask what the "practical significance" of the results are and you will sometimes hear this quantified with effect sizes or the raw descriptive statistics from a study. For example, let's say we have a control group and an experimental group of students. The treatment group is given a reading intervention. After the experiment, reading scores are observed and we find the result is statistically significant. However, we obtain the Cohen's d, which is approximately .01. This means that the treatment, though statistically significant, only yields less than a .01 standard deviation change in reading scores. As an educator, we would normally not care about this, so we would deem it to not be practically significant.

However, let's say we switched the situation and this Cohen's d score is for a new medicine about to hit the market. That .01 SD may matter, as it could literally mean the life or death of thousands of people (depending on how many people it is given to). Here the practical significance may change a lot.

Example

Here is a simulated example in R. Here I have two groups, an experimental and treatment group. After testing, their means are $\bar{x}_1 = 0$ and $\bar{x}_2 = .0009$, with both $\rm SD = .01$ and $n=1000$. Because of the large sample size and low standard error that results from the small standard deviation, we inevitably get a significant t-test despite the miniscule difference in means.

#### Setup Data ####
library(tidyverse)
library(effectsize)
set.seed(123)
Control <- rnorm(n=1000, mean=0, sd=.01)
Treatment <- rnorm(n=1000, mean=.0009, sd=.01)

df <- data.frame(Control,Treatment) %>% 
  gather() %>%  
  rename(Group = key,
         Value = value) %>% 
  as_tibble()
df

#### T-Test ####
t <- t.test(Value ~ Group, data = df)
t

#### Cohen's D ####
d <- cohens_d(Value ~ Group, data = df)
d

As shown by the result from t below, we get a statistically significant test:

    Welch Two Sample t-test

data:  Value by Group
t = -2.5995, df = 1997.4, p-value = 0.009405
alternative hypothesis: true difference in means between group Control and group Treatment is not equal to 0
95 percent confidence interval:
 -0.002041064 -0.000285684
sample estimates:
  mean in group Control mean in group Treatment 
           0.0001612787            0.0013246525  

If we obtain the Cohen's $d$ metric by printing d, we will find the effect size is quite weak:

Cohen's d |         95% CI
--------------------------
-0.12     | [-0.20, -0.03]

- Estimated using pooled SD.

We can visualize these small differences with a plot of the densities and their respective means for each group with some vertical red lines:

#### Plot ####
df %>% 
  ggplot(aes(x=Value,
             fill=Group)) +
  geom_density(linewidth=1,
               alpha = .4) +
  theme_minimal() +
  labs(x="Score",
       y="Density",
       title = "Experiment") +
  geom_vline(xintercept = mean(Control),
             linetype = "dashed",
             color="red") +
  geom_vline(xintercept = mean(Treatment),
             linetype = "dashed",
             color="red")

As you can see, there is hardly any realistic difference between these groups despite having statistically significant t-statistics.

enter image description here

Keep in mind that the raw estimates matter here. A decimal difference may not matter in terms of educational scores (such as an ACT/SAT score) but may matter a lot for something else (such as a blood alcohol content level).

Edit

What you have stipulated in the comments is unfortunately commonly asserted but nonetheless untrue. If you go back and read Sir Ronald Fisher's book The Design of Experiments (one of the books that introduced NHST), he even explicitly states this on Pages 18-19, where he says you can only reject the null.

enter image description here

enter image description here

Citation

Fisher, R. A. (1974). The design of experiments (9. ed). Hafner Press.

$\endgroup$
20
  • 1
    $\begingroup$ Did you have a question about this statement? $\endgroup$ Commented Aug 31, 2023 at 6:22
  • 1
    $\begingroup$ Care to explain why? $\endgroup$ Commented Sep 4, 2023 at 11:46
  • 1
    $\begingroup$ @SubhashC.Davar practical significance refers to the effect size of a result. It has nothing to do with p-values. Shawn's sentence you quote is a literal example of a result having low practical significance (despite being statistically significant) because a change of .01 SD in reading scores will be unnoticeable to teachers and students. How is that not valid? $\endgroup$
    – Sointu
    Commented Sep 4, 2023 at 12:46
  • 2
    $\begingroup$ Nobody said Cohen's d measures statistical significance. In terms of practical significance, lets say we have a group whose average tests scores are 70 with a SD of 15. If we perform an intervention on another group with this .01 SD difference, their average score would only improve to 70.15. That is essentially meaningless. So yes, this is practical, because I as a teacher would never waste time and resources on an intervention that has no practical benefit to my students. $\endgroup$ Commented Sep 4, 2023 at 14:18
  • 2
    $\begingroup$ The outcome of the test in Shawn's education example (statistical significance) will be important to the researcher (otherwise why undertake it). By stating that the (point) effect estimate is not viewed by the researcher as large enough to be worth actioning in some way is simply a comment about the opportunities available if the estimated effect was the true effect. Other readers may disagree with the researcher regarding what is or is not educationally significant. Also, remember that the researcher's comment is not a confidence claim about the size of the true effect. $\endgroup$ Commented Sep 4, 2023 at 17:44
9
$\begingroup$

The difference is magnitude of evidence vs magnitude of effect

A simple way to understand this is to remember that "statistical significance" refers to the magnitude of evidence (typically of a non-zero effect), whereas "practical significance" refers to the magnitude of the effect. The distinction exists because it is possible that there could be a large amount of evidence for a small effect, or that there could be a small amount of evidence of a large effect, etc. In general, confusion between statistical and practical significance is an instance of the quantifier-shift fallacy where the quantifier of "large" or "high magnitude" is shifted from the evidence size to the effect size, or vice versa.

Here is a useful way that I explain this distinction to students, using a legal analogy. Suppose you consider a situation where a person is driving 5km/h over the speed limit and they drive through a speed camera. Let us stipulate that this speed camera is highly effective in tracking their speed at it gives highly reliable evidence of their actual speed, within a very small tolerance (small enough to say that they were speeding with a high level of confidence). In this case, would you explain this situation by saying "There is significant evidence of a crime", or would you say "There is evidence of a significant crime"? If you said the latter you would sound a bit silly because speeding in your car is certainly not a "significant crime" --- rape, murder, arson are "significant crimes", but low-level speeding is a very minor crime. The latter would constitute an instance of the quantifier-shift fallacy where you incorrectly present a high-magnitude of evidence for a low-magnitude crime as if it were evidence of a high-magnitude crime.

As you can see from the above, the concept of "statistical significance" is not wrongly conceived. Both statistial significance and practical significance are useful concepts and should typically be considered together. They are distinct concepts because they each refer to a high-magnitude of different things (evidence versus effect size). Good statistical analysis ---like good legal analysis--- retains the distinction between the magnitude of evidence for a non-zero effect, and the magnitude of the effect itself. Both are important in many scientific and practical contexts and both should be understood well for a practicing statistician.

$\endgroup$
8
$\begingroup$

The term "practical significance" is used by some researchers to denote that the estimated effect is of a meaningful size in that particular application context. There is no clash with the concept of statistical significance. For example, the "drug B's estimated effect of 0.2 is both statistically and practically significant".

The alternative others prefer to use to denote the same judgement and result is to be more direct and say "drug B's estimated effect is both medically and statistically significant". On other occasions when the estimated effect is too small to be meaningful, the researcher might state that the estimated effect was statistically significant but not medically significant. Providing both types of information leaves the reader better informed.

In other fields, you could report whether or not an effect is biologically significant, geologically significant, financially significant, etc, etc.

$\endgroup$
4
  • $\begingroup$ please avoid confusing statements e.g. - "statistically and practically" $\endgroup$ Commented Aug 31, 2023 at 6:41
  • 1
    $\begingroup$ Exactly, that is another reason why I prefer the direct method! $\endgroup$ Commented Aug 31, 2023 at 6:44
  • $\begingroup$ statistically significant but not medically significant. ? That is obscure & needs explanatory note. $\endgroup$ Commented Sep 5, 2023 at 5:12
  • $\begingroup$ "statistically and practically significant": Adding the word practically results in confusion and distraction in interpretation. $\endgroup$ Commented Sep 5, 2023 at 23:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.