When doing linear regression, all sorts of influence checks (Cook's Distance, leverage, dffits, dfbetas, covratio) can be conducted on the data points. Each of these are literature-supplied with some cut-off levels, i.e. Cook's D > 4/n. However, I am concerned with a special case: When a single data point drives the significance under the magic p=0.05 value. I see this in many life science regression plots. Not only the omission of a single data point would revert a regression from "significant" to "insignificant", but also small shifts of a single value can do this. Consider the following:

## Create significant model
a <- 1:20
b <- 5 + 0.08 * a + rnorm(20, 0, 1)
LM <- lm(b ~ a)

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  5.01645    0.48052   10.44 4.59e-09 ***
a            0.09667    0.04011    2.41   0.0269 *  

Residual standard error: 1.034 on 18 degrees of freedom
Multiple R-squared:  0.2439,    Adjusted R-squared:  0.2019 
F-statistic: 5.807 on 1 and 18 DF,  p-value: 0.02688

If we simply exchange $b_1$ from its value (6.013) to 6.7, i.e. a mere shift of 10% of a single value out of 20, we obtain:

b2 <- b
b2[1] <- 6.7
LM2 <- lm(b2 ~ a)

           Estimate Std. Error t value Pr(>|t|)    
(Intercept)  5.15379    0.50033   10.30 5.65e-09 ***
a            0.08686    0.04177    2.08   0.0521 .  

Residual standard error: 1.077 on 18 degrees of freedom  
Multiple R-squared:  0.1937,    Adjusted R-squared:  0.1489 
F-statistic: 4.325 on 1 and 18 DF,  p-value: 0.05213

Now researchers that are inclined to strictly adhere to the p=0.05 threshold might derive conclusions driven by a single datapoint. And IMHO, this shows that the data structure is highly unstable w.r.t. to significance. Furthermore, all classical influence measures fail to identify this data point #1 as an "outlier":

> influence.measures(LM2)
Influence measures of
lm(formula = b2 ~ a) :

     dfb.1_    dfb.a   dffit cov.r   cook.d    hat inf
1   0.74440 -0.63695  0.7451 1.053 2.57e-01    0.1857    

One can (by optimization) find, for each value, the "trust region" in which the complete regression stays significant (blue lines). Shifting any values (black points) outside any of these regions reverts significance, either by influencing the slope or its standard error in way that pt(slope/se(slope), n-2) > 0.05. trust regions Question would be: Is this interesting stuff? Or am I missing some obvious and rather boring facts?

  • $\begingroup$ This is definitely interesting, but it would be of more practical use if you could somehow translate the regions to a p-value sensitivity. For example, y-axis could be change in absolute y value divided by change in p-value. $\endgroup$ – Jean-Paul Feb 11 '17 at 8:25
  • $\begingroup$ You might be interested in techniques which perturb the original data to see the effect on the model. There is, for instance, and R package called perturb which does this. Its design goal is to see the effect of collinearity but it might also help for some of your use cases. Or perhaps not. $\endgroup$ – mdewey Feb 11 '17 at 14:09

I am on the side of saying this is merely another way of understanding deletion diagnostics. Perturbations of a point are highly related to that point's influence function which is also estimated by deletion diagnostics. Typically the df-betas are used to show deletion diagnostics, but they can be scaled to be standardized df-betas and then compared to critical values for their approximate N(0,1) distribution.

In terms of perturbations of a point, perturbing it to conform with the line of best fit obtained by deleting that point has essentially the same impact on regression and inference as deleting that point altogether. It goes further to note that, for linear regression, any point not lying on the centroid of the predictor scale can be perturbed arbitrarily to achieve a particular slope or p-value. So we are at a loss of conceptualizing what a meaningful difference or perturbation might be, aside from reproducing some diagnostics that are already established.

  • $\begingroup$ The deletion effect on the p-value crossing its magical 0.05 border could possibly be calculated by using dfbetas (slope effect) and covratio (variance effect), but I don't really know how. However, I had the impression that none of the established influence measures (or combinations thereof) quantifies the effect on the slope's p-value, which is really often used as an argument for significant correlations in life science research. And would it not be too adventurous to state that significance should not depend solely on the presence (or location) of a single datapoint? $\endgroup$ – anspiess Dec 28 '17 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.