In a nutshell, what would be the main differences between practical and statistical significance? What would be the relative importance of each in drawing scientific inferences, for example?
-
5$\begingroup$ statistical-significance is well-defined and easy to look up. Practical significance: not so much. $\endgroup$– Nick StaunerJan 10, 2014 at 1:17
-
1$\begingroup$ Practical significance depends on the particular field of study. You can't arbitrarily say that +/- 5% change is practically significant across all fields. In chem/bio it may be the difference between life and death, but in finance or geology that change may not mean very much at all. Consult discipline-specific opinions in past studies/articles to determine what other people view as practical significance. $\endgroup$– rocinanteJan 10, 2014 at 2:39
8 Answers
In a nutshell:
Suppose you are testing the effect of drinking a glass of wine per day on the risk of a heart attack. In layman terms, statistical significance refers to whether drinking affects such risk. Practical significance refers to whether such change in risk is of a relevant magnitude for real life concerns. If this is not the case, then you should not be worried about drinking one glass of wine per day, as the effect it has on your health is too small to be a concern.
More precisely, say you find, for a given significance level, that the effect of drinking is non-zero (e.g. p-value below 0.05). Then, the effect of wine on risk of heart attack is statistically significant.
Say however, that such effect, albeit non-negative is "very small". For example, say that drinking one glass of wine per day increases your risk of heart attack by 0.01%. Then, such effect has no practical significance. Conversely, if such effect were to be "large" (e.g. a 20% increase), the effect has practical significance. (the qualification of "very small" and "large" is usually estimated based on the standard deviation of the outcome variable in the sample/population)
In economics, the term for the latter is "economic significance". It is common to ask, after finding a statistically significant effect, whether such effect is economically significant. This is sometimes seen by exploring how much one standard deviation change in the regressor/control relates to the distribution of the dependent/outcome variable.
-
$\begingroup$ (-1) Your explanation of practical sig. is pretty on-target, but not that of statistical sig. Cause and effect is a different matter. You say correctly that statistical sig. involves p-values, but I'm not seeing any helpful info beyond that here. $\endgroup$– rolando2Sep 17 at 1:56
-
$\begingroup$ Agree there's a misleading focus on cause and effect here. Statistical significance makes no claims whatsoever about directionality of associations, we cannot claim that statistical association between two variables implies that either one "affects" the other. Ice cream sales and shark attacks show significant association simply because both peak in the summer; that doesn't imply you can reduce shark attacks by curtailing ice cream sales. In your wine example, a statistical association alone says little about whether changing your wine consumption will actually change your heart attack risk. $\endgroup$ Sep 18 at 13:52
Some points about the practical importance (I'm trying to avoid significant here) of statistically significant findings that come to my mind:
With increasing sample size, small effects will become statistically significant. Whether they are practically of importance won't change.
This is another way of looking at @Adam Bailey's point.On top of that, I see a second level of practical significance: Roughly speaking, "statistically significant" means it is unlikely to observe such (or more extreme) differences if the null hypothesis is true. This is also the case for the carefully formulated Null hypothesis that takes into account the effect size that would be considered as practically important according to the previous point.
However, usually this is not all that I as your reader am interested in. What I want to know is not only what your findings are but also how much I can rely on your conclusions. That would be more like the predictive value of your results. In order to judge this, I'd need to know the "prevalence" of true hypotheses among the hypotheses you generate and test.
Which is a way of saying that testing a huge number of more-or-less randomly generated hypotheses usually is not going to help much, and why the total number of tests conducted is of huge importance for the practical conclusions you can draw from a statistically significant result.
All the answers are great. But I'll add one more point. Because the word "significant" has two meanings that can both apply when analyzing data, it is best to avoid that word entirely. Its use has caused too much confusion.
If you want to talk about p-values, state the value. Or state the threshold it is less than. The S-word is not needed.
If you want to talk about the size of the effect, do so, in its own units (with confidence intervals). If you want to describe importance or impact or consequences or ramifications ... use words like that. The S-word is not needed.
-
1$\begingroup$ (-1) A. "confusion (not among statisticians [...])": if you look up the work of Michael Oakes or Brian Everitt or Gerd Gigerenzer, you'll see abundant evidence that statisticians, analysts, and researchers, just like lay people, tend to interpret results incorrectly when it comes to statistical sig. Not that I haven't made these mistakes myself. B. To say merely whether a p-value met some threshold is nowhere near sufficient in most situations. This is discussed extensively in the ASA's special publications on p-values from 2016. amstat.org/asa/files/pdfs/P-ValueStatement.pdf $\endgroup$– rolando2Sep 17 at 2:01
-
1$\begingroup$ @rolando2. I took out the phrase "not among statisticians", which wasn't helpful or detailed. I agree with you, of course that statistical hypothesis testing is overused (and misunderstood), but that is really not the point of the question or my answer. $\endgroup$ Sep 18 at 13:47
This is a large topic but I suggest that the following are key points.
The relation between practical and statistical significance is not well described in terms of relative importance. To think in those terms rather suggests that a researcher has collected some data, used the data to test a hypothesis chosen without too much thought (perhaps that a regression parameter differs from zero), found that the test points to rejection of the hypothesis, and then asked whether this finding is significant. Having reached that position it is certainly appropriate to assess practical significance, rather than assuming that statistical significance implies practical significance.
However, a better approach in the context of scientific research is to consider practical significance much earlier, at the stage of research design. A hypothesis can then be chosen such that whether or not it is found to be correct is of practical significance. In the case of a regression parameter $\beta$, for example, this might lead to the choice of a null hypothesis as $H_0: \beta > \delta$, for some value $\delta$ judged to be a threshold of practical significance, rather than simply $H_0: \beta = 0$. Of course the appropriate value of $\delta$ will depend as rocinante says on the field of study, and also on the particular question within that field. Once a hypothesis is selected, consideration can then be given to data collection, and having collected data the hypothesis can be tested, the result being assessed in terms of statistical significance.
-
$\begingroup$ Not that I don't agree with the spirit of your argument, however thoughtlessness is not really the reason you're testing whether a coefficient is different from zero. It's precisely because statistical and practical significance are two different things that these thresholds exist. You run the risk of dismissing a regressor out of hand when your threshold is practical significance. The wording of such conclusions requires careful consideration so that readers don't get the wrong impression of what you're actually saying. $\endgroup$ Jan 10, 2014 at 12:08
Statistical significance always becomes more likely the larger the sample size is. This holds since almost no experiment is done exactly under the null hypothesis. Small irrelevant differences can never be ruled out. Consistency of the tests can make such irrelevant differences significant. Practical relevance depends on the scientific fact, not on statistical convergences. In applied sciences, effect sizes are irrelevant if they cause no difference in the expert's judgement.
But both concepts can be brought together: Statistical tests for relevance can often be constructed by choosing an interval of irrelevant effect sizes. Significant relevance at $\alpha$ level can be concluded, iff a $1-\alpha$-confidence interval is disjoint from this irrelevance interval. This way, enlarging the sample size increases the chance of statistical significance iff the effect is relevant. Otherwise even a confidence interval of length $0$ stays inside the interval of irrelevance.
Nutshell is being studied in relation to, among other things, prevention of diabetes and obesity.
Say you wish to analyse the difference between rats that get nutshell extract in their feed and rats that do not.
- Then the 'effect of the nutshell' is statistically significant if the results are unlikely given the hypothesis that there is no effect of the nutshell.
- And the 'effect of the nutshell' is practically significant if the results show a large size difference (estimated), 'large' being defined in relation to practical purposes.
Importance in drawing scientific inference: Statistical and practical significance do not need to occur at the same time (depending on the test size and variation of the population). It depends much on the estimated standard error for the effect size. If the SE is small then very tiny (practically insignificant) effects can still be tested as statistically significant. And if the SE is big then large (practically significant) effects can be tested as statistically insignificant (not because the effect is small but because the test is a bad detector).
So errors are being made if these different types of significance are mixed:
- Arguing that no effect is present because of low statistical significance, while the test might have been not accurate enough to detect a practical significant difference.
- Arguing that a (practical )significant effect is present because of high significance, while the test might have been extremely sensitive to detect tiny effect sizes with little practical significance.
-
$\begingroup$ Where I say "statistically significant if the results are unlikely", I should emphasize that this relates to the 'likelihood is low'. Since, from a Bayesian perspective, a low significance level, say 5% does not mean directly that the probability of the $H_0$ being true is equally low (or probability of the $H_0$ being false is high). A very interesting example is the court case in this link math.leidenuniv.nl/~gill/lucia.html $\endgroup$ Sep 25, 2017 at 16:02
The problem exists only if you don't use Bayesian inference. Bayesian posterior distributions completely cut through these confusing issues. As an example suppose that one is interested in quantifying evidence that a new drug lowers blood pressure. Let $\Delta$ be the parameter in the regression model associated with the difference in mean systolic blood pressure (SBP). A Bayesian regression model (which would also handle random effects without any approximations needed, by the way) gives rise to a posterior distribution for $\Delta$. Suppose that the minimum clinically important difference in SBP is 7 mmHg (the difference one would really not want to miss; the basis for Bayesian or frequentist power calculations) and that the minimum detectable change is 3 mmHg, i.e., we consider that a change in mean SBP < 3 is trivial. Then one simply computes
- $\Pr(\Delta > 0)$
- $\Pr(\Delta > 3)$
- $\Pr(\Delta > 7)$
and likewise for all possible values of $\Delta$. Bayes provides evidence in favor of any interval of $\Delta$ whereas frequentist inference mainly provides evidence against one particular $\Delta$ (zero).
In interpreting the study I'd concentrate of the first and second posterior probabilities, which represent evidence for any SBP reduction and evidence for a non-trivial reduction.
That's how Bayesian inference deals with practical significance.
-
1$\begingroup$ Helpful, but I think it'd be even more helpful for more readers if you could reword a bit to clarify how <7 is not a "clinically important difference" but only those values <3 are "trivial." I think you mean "trivial" referring to a matter of ability to measure rather than the importance of a given effect, the latter being closer to the topic at hand. $\endgroup$– rolando2Sep 17 at 2:16
-
2$\begingroup$ Points definitely worth considering. I was making an assumption about "7" being chosen by hypertension experts. I like to think of half of the minimally clinically important difference as being a threshold for triviality, and am motivated by the fact that most false rejections of $H_0$ in frequentism come from small observed effects with p-values just above $\alpha$. $\endgroup$ Sep 17 at 11:29
While I agree with the existing answers, I think one nuance hasn't been stressed enough and I will try to illustrate it with an example.
If you increase $n$, the power of your test rises which means that you are better able to detect real effects. If your $n$ is very large, you will be able to detect very small effects which are practically irrelevant.
Although not of practical relevance, those tiny effects that you can detect with very large $n$ are still real though. They are not statistical flukes as they are sometimes mistakenly called. A large $n$ doesn't give you an increased risk of finding small effects that do not exist. If such was the case, we would need to warn against studies with large $n$. The detected small effects are real, but unimportant for reasons related to practical applications.
Let's suppose you measure the height of one million male babies, all exactly one year old since their birth and none of which were prematurely born. One week later, you measure the same million babies again.
Usually, I would advice to use two-sided tests, but here we can exclude that the babies have shrunk I would say:
- $H_0$ The babies have not grown
- $H_a$ The babies have grown
You can, for each baby, subtract both measures and see how much it has grown. This gives you one column with a million measures. A t-test can show if it's mean is significantly larger than 0. The enormous sample size of 1000000 and the paired setup increase the power of the test.
According to the WHO the median baby grows from 75.7cm to 87.8cm between his first and second birthday. Assuming linearity within that year, babies would grow on average $0.23cm$ during the week that our observations span.
With the enormous $n$, I am quite confident we could find a highly statistically significant result that the growth was not zero, a very low p-value. That growth of the average baby is real, not a statistical fluke stemming from our large sample size. But does that mean this result has practical relevance? No, as a clothing manufacturer I still wouldn't produce separate onesies for 53 week olds than for 52 week olds for two reasons:
- 0.23cm is not enough of a difference to warrant another intermediate size of clothing.
- Those 0.23cm are for the average baby anyway, not for each individual baby. Individual shoppers are used buying smaller or larger sizes than indicated for the age. Because the variance of baby height is considerable and doesn't depend in any way on our study, some parents don't have the choice but to do this. This variance of baby height is not to be confused with the variance of the estimator of the average baby height, that one decreases with $n$, hence the power of the test.
In this example, the scale on which the test compares measures is intuitive and the effect size can and should be measured with a confidence interval on that scale. In some scenarios that cannot be done, for example when comparing the effect sizes of multiple tests on different scales. Standardized effect size measures like Cohen's d can then be used.
Simulation in R:
I cannot simulate what it would look like to measure the same babies twice, but I can reasonably approximate a test in which a million 52 week olds were measured and another million other babies which are all 53 weeks old were measured for comparison.
This test is not paired and has lower power than the paired one. If I can show that the practically irrelevant $0.23cm$ are statistically significant with this test, they will be significant with the other as well.
Height is normally distributed and from the WHO data, I can roughly deduce the following distributions:
- 52 week olds: $\mathcal{N}(\mu=75.735,\sigma=(80.35 - 71.10)/4=2.313)$
- 53 week olds: $\mathcal{N}(\mu=75.995,\sigma=(80.79 - 71.25)/4=2,385)$
I did this using the thumb rule that there are 4 standard deviations $\sigma$ between the $0.025^{th}$ and the $0.975^{th}$ quantile.
> t.test(rnorm(1e+6,75.995,2.385),
rnorm(1e+6,75.735,2.3125),
alternative = "two.sided",
mu = 0,
paired = FALSE,
var.equal = FALSE,
conf.level = 0.999)
Welch Two Sample t-test
data: rnorm(1e+06, 75.995, 2.385) and rnorm(1e+06, 75.735, 2.3125)
t = 79.403, df = 1998100, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
99.9 percent confidence interval:
0.2528959 0.2747625
sample estimates:
mean of x mean of y
75.99828 75.73445
I even used the Welch test that doesn't assume equal variances (they are very close to equal) and used a two sided test (babies will not shrink). Both of these circumstances reduce the power of the test. And yet the statistical significance of the test is exceptional with $p-value < 2.2 * 10^{-16}$.
Regarding the effect size we can say that 999 out of 1000 confidence intervals would contain the true height gain of the average baby and our confidence interval is $[ 0.253, 0.275 ]$ cm which is a practically irrelevant although real quantity.
-
2$\begingroup$ I think your conclusion is valid in a model-truth world. In a world where models are approximations, maybe not $\endgroup$ Sep 23, 2017 at 14:03
-
$\begingroup$ Why would the model being an approximation of reality influence the question whether small effects, that can only be detected by doing numerous measures, are still real effects? There could of course be scenarios where there is s small but systematic bias that only becomes relevant compared to those small effect and that you would otherwise never see (i.e. a different instrument, biased compared to the first, was used to measure all babies the second time). That's still not a statistical fluke though, it's a systematic effect (though an erroneous one) that was in the data you gave to the test. $\endgroup$ Sep 23, 2017 at 14:28
-
$\begingroup$ I would call it a statistical fluke if some random sampling error gets mistaken for a fixed effect. True the measurements of 52 and 53 week olds, or some other measurement might be (statistically) different. Yet, it is not necessarily the effect of growth. And I believe that it becomes more likely to make these mistakes as the effect sizes get bigger. That is because the experiments in the real world are not like a random number generator. (and then I do not even consider errors of the statistical models, for instance what do round-off errors do to the assumptions?) $\endgroup$ Sep 25, 2017 at 16:18
-
$\begingroup$ But it is not a random sampling error. It couldn't be a random sampling error with these sizes of $n$. If it's not a random sampling error, neither according to you an effect of growth, what else do you suppose happened? What do you mean putting only the word effect in cursive? How could it be growth without an effect of growth when growth is simply defined as the effect of interest? It could be a systematic measurement bias between both samples. In this case, make sure your instruments have biases smaller than the effect sizes you are interested in, or preferably use the same instrument. $\endgroup$ Sep 25, 2017 at 16:50
-
$\begingroup$ You might have random effects $y_{ijkl} = a_i + b_j + c_k + \epsilon_l$, with $a_i$ effect for week older, $b_j$ random effect for individuals, $c_k$ random effect for other classes (e.g. the group number, or day of measurement, ...), and $\epsilon_l$ a measurement error. If you only measure 2 times, then your $j$ and $l$ might be many, but your classes $k$ are small. Also, they correlate with the effect $i$. Often effects $c_k$ are neglected, assuming the $c_k$ that may correlate with the $a_i$ are tiny or corrected. Yet, if we see a tiny effect, then this assumption may not be true. $\endgroup$ Sep 25, 2017 at 18:24