Practical vs Statistical significance

Question

In a nutshell, what would be the main differences between practical and statistical significance? What would be the relative importance of each in drawing scientific inferences, for example?

Answer 1

In a nutshell:

Suppose you are testing the effect of drinking a glass of wine per day on the risk of a heart attack. In layman terms, statistical significance refers to whether drinking affects such risk. Practical significance refers to whether such change in risk is of a relevant magnitude for real life concerns. If this is not the case, then you should not be worried about drinking one glass of wine per day, as the effect it has on your health is too small to be a concern.

---

More precisely, say you find, for a given significance level, that the effect of drinking is non-zero (e.g. p-value below 0.05). Then, the effect of wine on risk of heart attack is statistically significant.

Say however, that such effect, albeit non-negative is "very small". For example, say that drinking one glass of wine per day increases your risk of heart attack by 0.01%. Then, such effect has no practical significance. Conversely, if such effect were to be "large" (e.g. a 20% increase), the effect has practical significance. (the qualification of "very small" and "large" is usually estimated based on the standard deviation of the outcome variable in the sample/population)

In economics, the term for the latter is "economic significance". It is common to ask, after finding a statistically significant effect, whether such effect is economically significant. This is sometimes seen by exploring how much one standard deviation change in the regressor/control relates to the distribution of the dependent/outcome variable.

Answer 2

Some points about the practical importance (I'm trying to avoid significant here) of statistically significant findings that come to my mind:

• With increasing sample size, small effects will become statistically significant. Whether they are practically of importance won't change.
  This is another way of looking at @Adam Bailey's point.

• On top of that, I see a second level of practical significance: Roughly speaking, "statistically significant" means it is unlikely to observe such (or more extreme) differences if the null hypothesis is true. This is also the case for the carefully formulated Null hypothesis that takes into account the effect size that would be considered as practically important according to the previous point.
  However, usually this is not all that I as your reader am interested in. What I want to know is not only what your findings are but also how much I can rely on your conclusions. That would be more like the predictive value of your results. In order to judge this, I'd need to know the "prevalence" of true hypotheses among the hypotheses you generate and test.
  Which is a way of saying that testing a huge number of more-or-less randomly generated hypotheses usually is not going to help much, and why the total number of tests conducted is of huge importance for the practical conclusions you can draw from a statistically significant result.

Answer 3

All the answers are great. But I'll add one more point. Because the word "significant" has two meanings that can both apply when analyzing data, it is best to avoid that word entirely. Its use has caused too much confusion.

If you want to talk about p-values, state the value. Or state the threshold it is less than. The S-word is not needed.

If you want to talk about the size of the effect, do so, in its own units (with confidence intervals). If you want to describe importance or impact or consequences or ramifications ... use words like that. The S-word is not needed.

Answer 4

This is a large topic but I suggest that the following are key points.

The relation between practical and statistical significance is not well described in terms of relative importance. To think in those terms rather suggests that a researcher has collected some data, used the data to test a hypothesis chosen without too much thought (perhaps that a regression parameter differs from zero), found that the test points to rejection of the hypothesis, and then asked whether this finding is significant. Having reached that position it is certainly appropriate to assess practical significance, rather than assuming that statistical significance implies practical significance.

However, a better approach in the context of scientific research is to consider practical significance much earlier, at the stage of research design. A hypothesis can then be chosen such that whether or not it is found to be correct is of practical significance. In the case of a regression parameter $\beta$, for example, this might lead to the choice of a null hypothesis as $H_0: \beta > \delta$, for some value $\delta$ judged to be a threshold of practical significance, rather than simply $H_0: \beta = 0$. Of course the appropriate value of $\delta$ will depend as rocinante says on the field of study, and also on the particular question within that field. Once a hypothesis is selected, consideration can then be given to data collection, and having collected data the hypothesis can be tested, the result being assessed in terms of statistical significance.

Answer 5

Statistical significance always becomes more likely the larger the sample size is. This holds since almost no experiment is done exactly under the null hypothesis. Small irrelevant differences can never be ruled out. Consistency of the tests can make such irrelevant differences significant. Practical relevance depends on the scientific fact, not on statistical convergences. In applied sciences, effect sizes are irrelevant if they cause no difference in the expert's judgement.

But both concepts can be brought together: Statistical tests for relevance can often be constructed by choosing an interval of irrelevant effect sizes. Significant relevance at $\alpha$ level can be concluded, iff a $1-\alpha$-confidence interval is disjoint from this irrelevance interval. This way, enlarging the sample size increases the chance of statistical significance iff the effect is relevant. Otherwise even a confidence interval of length $0$ stays inside the interval of irrelevance.

Answer 6

Nutshell is being studied in relation to, among other things, prevention of diabetes and obesity.

Say you wish to analyse the difference between rats that get nutshell extract in their feed and rats that do not.

• Then the 'effect of the nutshell' is statistically significant if the results are unlikely given the hypothesis that there is no effect of the nutshell.
• And the 'effect of the nutshell' is practically significant if the results show a large size difference (estimated), 'large' being defined in relation to practical purposes.

Importance in drawing scientific inference: Statistical and practical significance do not need to occur at the same time (depending on the test size and variation of the population). It depends much on the estimated standard error for the effect size. If the SE is small then very tiny (practically insignificant) effects can still be tested as statistically significant. And if the SE is big then large (practically significant) effects can be tested as statistically insignificant (not because the effect is small but because the test is a bad detector).

So errors are being made if these different types of significance are mixed:

• Arguing that no effect is present because of low statistical significance, while the test might have been not accurate enough to detect a practical significant difference.
• Arguing that a (practical )significant effect is present because of high significance, while the test might have been extremely sensitive to detect tiny effect sizes with little practical significance.

Answer 7

The problem exists only if you don't use Bayesian inference. Bayesian posterior distributions completely cut through these confusing issues. As an example suppose that one is interested in quantifying evidence that a new drug lowers blood pressure. Let $\Delta$ be the parameter in the regression model associated with the difference in mean systolic blood pressure (SBP). A Bayesian regression model (which would also handle random effects without any approximations needed, by the way) gives rise to a posterior distribution for $\Delta$. Suppose that the minimum clinically important difference in SBP is 7 mmHg (the difference one would really not want to miss; the basis for Bayesian or frequentist power calculations) and that the minimum detectable change is 3 mmHg, i.e., we consider that a change in mean SBP < 3 is trivial. Then one simply computes

• $\Pr(\Delta > 0)$
• $\Pr(\Delta > 3)$
• $\Pr(\Delta > 7)$

and likewise for all possible values of $\Delta$. Bayes provides evidence in favor of any interval of $\Delta$ whereas frequentist inference mainly provides evidence against one particular $\Delta$ (zero).

In interpreting the study I'd concentrate of the first and second posterior probabilities, which represent evidence for any SBP reduction and evidence for a non-trivial reduction.

That's how Bayesian inference deals with practical significance.

Answer 8

While I agree with the existing answers, I think one nuance hasn't been stressed enough and I will try to illustrate it with an example.

If you increase $n$, the power of your test rises which means that you are better able to detect real effects. If your $n$ is very large, you will be able to detect very small effects which are practically irrelevant.

Although not of practical relevance, those tiny effects that you can detect with very large $n$ are still real though. They are not statistical flukes as they are sometimes mistakenly called. A large $n$ doesn't give you an increased risk of finding small effects that do not exist. If such was the case, we would need to warn against studies with large $n$. The detected small effects are real, but unimportant for reasons related to practical applications.

Let's suppose you measure the height of one million male babies, all exactly one year old since their birth and none of which were prematurely born. One week later, you measure the same million babies again.

Usually, I would advice to use two-sided tests, but here we can exclude that the babies have shrunk I would say:

• $H_0$ The babies have not grown
• $H_a$ The babies have grown

You can, for each baby, subtract both measures and see how much it has grown. This gives you one column with a million measures. A t-test can show if it's mean is significantly larger than 0. The enormous sample size of 1000000 and the paired setup increase the power of the test.

According to the WHO the median baby grows from 75.7cm to 87.8cm between his first and second birthday. Assuming linearity within that year, babies would grow on average $0.23cm$ during the week that our observations span.

With the enormous $n$, I am quite confident we could find a highly statistically significant result that the growth was not zero, a very low p-value. That growth of the average baby is real, not a statistical fluke stemming from our large sample size. But does that mean this result has practical relevance? No, as a clothing manufacturer I still wouldn't produce separate onesies for 53 week olds than for 52 week olds for two reasons:

1. 0.23cm is not enough of a difference to warrant another intermediate size of clothing.
2. Those 0.23cm are for the average baby anyway, not for each individual baby. Individual shoppers are used buying smaller or larger sizes than indicated for the age. Because the variance of baby height is considerable and doesn't depend in any way on our study, some parents don't have the choice but to do this. This variance of baby height is not to be confused with the variance of the estimator of the average baby height, that one decreases with $n$, hence the power of the test.

In this example, the scale on which the test compares measures is intuitive and the effect size can and should be measured with a confidence interval on that scale. In some scenarios that cannot be done, for example when comparing the effect sizes of multiple tests on different scales. Standardized effect size measures like Cohen's d can then be used.

Simulation in R:

I cannot simulate what it would look like to measure the same babies twice, but I can reasonably approximate a test in which a million 52 week olds were measured and another million other babies which are all 53 weeks old were measured for comparison.

This test is not paired and has lower power than the paired one. If I can show that the practically irrelevant $0.23cm$ are statistically significant with this test, they will be significant with the other as well.

Height is normally distributed and from the WHO data, I can roughly deduce the following distributions:

• 52 week olds: $\mathcal{N}(\mu=75.735,\sigma=(80.35 - 71.10)/4=2.313)$
• 53 week olds: $\mathcal{N}(\mu=75.995,\sigma=(80.79 - 71.25)/4=2,385)$

I did this using the thumb rule that there are 4 standard deviations $\sigma$ between the $0.025^{th}$ and the $0.975^{th}$ quantile.

> t.test(rnorm(1e+6,75.995,2.385),
    rnorm(1e+6,75.735,2.3125),
    alternative = "two.sided", 
    mu = 0, 
    paired = FALSE, 
    var.equal = FALSE, 
    conf.level = 0.999)

Welch Two Sample t-test

data:  rnorm(1e+06, 75.995, 2.385) and rnorm(1e+06, 75.735, 2.3125)
t = 79.403, df = 1998100, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
99.9 percent confidence interval:
0.2528959 0.2747625
sample estimates:
mean of x mean of y 
75.99828  75.73445 

I even used the Welch test that doesn't assume equal variances (they are very close to equal) and used a two sided test (babies will not shrink). Both of these circumstances reduce the power of the test. And yet the statistical significance of the test is exceptional with $p-value < 2.2 * 10^{-16}$.

Regarding the effect size we can say that 999 out of 1000 confidence intervals would contain the true height gain of the average baby and our confidence interval is $[ 0.253, 0.275 ]$ cm which is a practically irrelevant although real quantity.