Outliers in Linear Regression that ONLY revert significance

When doing linear regression, all sorts of influence checks (Cook's Distance, leverage, dffits, dfbetas, covratio) can be conducted on the data points. Each of these are literature-supplied with some cut-off levels, i.e. Cook's D > 4/n. However, I am concerned with a special case: When a single data point drives the significance under the magic p=0.05 value. I see this in many life science regression plots. Not only the omission of a single data point would revert a regression from "significant" to "insignificant", but also small shifts of a single value can do this. Consider the following:

## Create significant model
set.seed(125)
a <- 1:20
b <- 5 + 0.08 * a + rnorm(20, 0, 1)
LM <- lm(b ~ a)
summary(LM)

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  5.01645    0.48052   10.44 4.59e-09 ***
a            0.09667    0.04011    2.41   0.0269 *

Residual standard error: 1.034 on 18 degrees of freedom
Multiple R-squared:  0.2439,    Adjusted R-squared:  0.2019
F-statistic: 5.807 on 1 and 18 DF,  p-value: 0.02688


If we simply exchange $b_1$ from its value (6.013) to 6.7, i.e. a mere shift of 10% of a single value out of 20, we obtain:

b2 <- b
b2[1] <- 6.7
LM2 <- lm(b2 ~ a)
summary(LM2)

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  5.15379    0.50033   10.30 5.65e-09 ***
a            0.08686    0.04177    2.08   0.0521 .

Residual standard error: 1.077 on 18 degrees of freedom
Multiple R-squared:  0.1937,    Adjusted R-squared:  0.1489
F-statistic: 4.325 on 1 and 18 DF,  p-value: 0.05213


Now researchers that are inclined to strictly adhere to the p=0.05 threshold might derive conclusions driven by a single datapoint. And IMHO, this shows that the data structure is highly unstable w.r.t. to significance. Furthermore, all classical influence measures fail to identify this data point #1 as an "outlier":

> influence.measures(LM2)
Influence measures of
lm(formula = b2 ~ a) :

dfb.1_    dfb.a   dffit cov.r   cook.d    hat inf
1   0.74440 -0.63695  0.7451 1.053 2.57e-01    0.1857


One can (by optimization) find, for each value, the "trust region" in which the complete regression stays significant (blue lines). Shifting any values (black points) outside any of these regions reverts significance, either by influencing the slope or its standard error in way that pt(slope/se(slope), n-2) > 0.05. Question would be: Is this interesting stuff? Or am I missing some obvious and rather boring facts?

• This is definitely interesting, but it would be of more practical use if you could somehow translate the regions to a p-value sensitivity. For example, y-axis could be change in absolute y value divided by change in p-value. Commented Feb 11, 2017 at 8:25
• You might be interested in techniques which perturb the original data to see the effect on the model. There is, for instance, and R package called perturb which does this. Its design goal is to see the effect of collinearity but it might also help for some of your use cases. Or perhaps not. Commented Feb 11, 2017 at 14:09