What is meant by practical significance and statistical significance? How do we distinguish these concepts? [duplicate]

Question

I am unable to understand content reported in : Practical vs Statistical significance

Statistical significance-testing implies whether a sample-statistic matches with population estimate of effect-size.(mean,r or d etc.).It is based on sampling theory. An interpretation of the practical significance is vague and obscure. It could be that it reflects validity-coefficient as discussed in psychometric methods or something else.

My concern is how does it differ from the concept of "statistical" significance? And which of the two types of significance is better?

Answer 1

Explanation

The term "practical significance" is often an attempt to subdue the often erroneous usage of "significance" in statistics, which is supposed to be shorthand for "statistical significance." When an inferential test is "statistically significant," this is supposed to equate the probability of finding the result given from a test if the null hypothesis is true. If we surpass the alpha threshold usually given (in my field .05), we can reject the null hypothesis (that there is no effect). What this doesn't say at all is whether the null hypothesis is actually true or if the alternative hypothesis is consequently true either. There is a lot of literature about the problematic usage of null hypothesis significance testing (NHST), which you can read about here.

This is why sometimes you will hear people ask what the "practical significance" of the results are and you will sometimes hear this quantified with effect sizes or the raw descriptive statistics from a study. For example, let's say we have a control group and an experimental group of students. The treatment group is given a reading intervention. After the experiment, reading scores are observed and we find the result is statistically significant. However, we obtain the Cohen's d, which is approximately .01. This means that the treatment, though statistically significant, only yields less than a .01 standard deviation change in reading scores. As an educator, we would normally not care about this, so we would deem it to not be practically significant.

However, let's say we switched the situation and this Cohen's d score is for a new medicine about to hit the market. That .01 SD may matter, as it could literally mean the life or death of thousands of people (depending on how many people it is given to). Here the practical significance may change a lot.

Example

Here is a simulated example in R. Here I have two groups, an experimental and treatment group. After testing, their means are $\bar{x}_1 = 0$ and $\bar{x}_2 = .0009$, with both $\rm SD = .01$ and $n=1000$. Because of the large sample size and low standard error that results from the small standard deviation, we inevitably get a significant t-test despite the miniscule difference in means.

#### Setup Data ####
library(tidyverse)
set.seed(123)
Control <- rnorm(n=1000,mean=0,sd=.01)
Treatment <- rnorm(n=1000,mean=.0009,sd=.01)

df <- data.frame(Control,Treatment) %>% 
  gather() %>%  
  rename(Group = key,
         Value = value) %>% 
  as_tibble()
df

#### T-Test ####
t <- t.test(df$Value ~ df$Group)
t

As shown by the result below, with a statistically significant test:

Welch Two Sample t-test

data:  df$Value by df$Group
t = -3.5218, df = 1998, p-value = 0.0004382
alternative hypothesis: true difference in means between group Control and group Treatment is not equal to 0
95 percent confidence interval:
 -0.0024552999 -0.0006988682
sample estimates:
  mean in group Control mean in group Treatment 
          -0.0003216818            0.0012554022 

If we obtain the Cohen's d metric, we will find the effect size is quite weak:

> d <- effectsize::cohens_d(df$Value ~ df$Group)
> d
Cohen's d |         95% CI
--------------------------
-0.16     | [-0.25, -0.07]

- Estimated using pooled SD.

We can visualize these small differences with a plot of the densities and their respective means for each group with some vertical red lines:

#### Plot ####
df %>% 
  ggplot(aes(x=Value,
             fill=Group))+
  geom_density(linewidth=1,
               alpha = .4)+
  theme_minimal()+
  labs(x="Score",
       y="Density",
       title = "Experiment")+
  geom_vline(xintercept = mean(Control),
             linetype = "dashed",
             color="red")+
  geom_vline(xintercept = mean(Treatment),
             linetype = "dashed",
             color="red")

As you can see, there is hardly any realistic difference between these groups despite having statistically significant t-statistics.

Keep in mind that the raw estimates matter here. A decimal difference may not matter in terms of educational scores (such as an ACT/SAT score) but may matter a lot for something else (such as a blood alcohol content level).

Edit

What you have stipulated in the comments is unfortunately commonly asserted but nonetheless untrue. If you go back and read Sir Ronald Fisher's book The Design of Experiments (one of the books that introduced NHST), he even explicitly states this on Pages 18-19, where he says you can only reject the null.

Citation

Fisher, R. A. (1974). The design of experiments (9. ed). Hafner Press.

Answer 2

The difference is magnitude of evidence vs magnitude of effect

A simple way to understand this is to remember that "statistical significance" refers to the magnitude of evidence (typically of a non-zero effect), whereas "practical significance" refers to the magnitude of the effect. The distinction exists because it is possible that there could be a large amount of evidence for a small effect, or that there could be a small amount of evidence of a large effect, etc. In general, confusion between statistical and practical significance is an instance of the quantifier-shift fallacy where the quantifier of "large" or "high magnitude" is shifted from the evidence size to the effect size, or vice versa.

Here is a useful way that I explain this distinction to students, using a legal analogy. Suppose you consider a situation where a person is driving 5km/h over the speed limit and they drive through a speed camera. Let us stipulate that this speed camera is highly effective in tracking their speed at it gives highly reliable evidence of their actual speed, within a very small tolerance (small enough to say that they were speeding with a high level of confidence). In this case, would you explain this situation by saying "There is significant evidence of a crime", or would you say "There is evidence of a significant crime"? If you said the latter you would sound a bit silly because speeding in your car is certainly not a "significant crime" --- rape, murder, arson are "significant crimes", but low-level speeding is a very minor crime. The latter would constitute an instance of the quantifier-shift fallacy where you incorrectly present a high-magnitude of evidence for a low-magnitude crime as if it were evidence of a high-magnitude crime.

As you can see from the above, the concept of "statistical significance" is not wrongly conceived. Both statistial significance and practical significance are useful concepts and should typically be considered together. They are distinct concepts because they each refer to a high-magnitude of different things (evidence versus effect size). Good statistical analysis ---like good legal analysis--- retains the distinction between the magnitude of evidence for a non-zero effect, and the magnitude of the effect itself. Both are important in many scientific and practical contexts and both should be understood well for a practicing statistician.

Answer 3

The term "practical significance" is used by some researchers to denote that the estimated effect is of a meaningful size in that particular application context. There is no clash with the concept of statistical significance. For example, the "drug B's estimated effect of 0.2 is both statistically and practically significant".

The alternative others prefer to use to denote the same judgement and result is to be more direct and say "drug B's estimated effect is both medically and statistically significant". On other occasions when the estimated effect is too small to be meaningful, the researcher might state that the estimated effect was statistically significant but not medically significant. Providing both types of information leaves the reader better informed.

In other fields, you could report whether or not an effect is biologically significant, geologically significant, financially significant, etc, etc.