2
$\begingroup$

Linear models are ubiquitous in economic, social, health and nutritional sciences and the starting point for much research and many articles.

However, there is a problem with linear models. When the true underlying model is more complex and not exactly linear, then the estimated coefficients might be counterintuitive and have an opposite sign in comparison to the coefficient in the underlying 'true' model.

An example of this effect is in an answer to this question Interpretation of the linear predictor of a OLS model on binomial data where a logistic model is fitted with a straight line.

example of coefficient becoming negative

  • The true underlying model has values with $z=1$ higher than values with $z=0$

  • The fitted linear model has the opposite. Values with $z=1$ are lower than values with $z=0$.

    The reason that the linear model does this is because $z$ and $x$ correlate and the values with $z=1$ occur mostly in the high range of $x$. In this range the linear model makes a large positive bias and this can be corrected with a negative coefficient for $z$.

    Often these models are "tested" based on statistical tests which only look at the variation due to sampling. But in this case the error occurs due to a specific intereaction with the bias of the model (that doesn't get tested with a statistical significance test). So these effects might slip through unnoticed.


This effect seems to me as a typical error. It seems alike to other statistical fallacies like the Yule-Simpson effect, the misuse and wrong interpretation of p-values, and the piranha problem. The effect might be less common than those three but it is similar in the problem originating from a naive approach.

Question: has this effect been described? Where and by whom? (and maybe it even has a name?)


Code for the model and image:

logis = function(x) {
  (1+exp(-x))^-1
}

### data and plot
set.seed(1)
n = 10^4
d = 1+rbinom(n,8,0.5)
x = runif(n,-d,d)
z = rbinom(n,1,prob = logis(x-4))
y = rbinom(n,1,logis(x+z))
plot(x,jitter(y,0.05), pch = 21, col = 1, bg = z*2, cex = 0.7, 
     xlab = "x", ylab = "y")

### true model line
xs = seq(-8,8,0.001)
lines(xs,logis(xs), lty = 2, lwd = 2)
lines(xs,logis(xs+1), lty = 2, lwd = 2, col = 2)


### linear model line fit
mod = lm(y~x+z)
mod
## result
# Call:
# lm(formula = y ~ x + z)
#
# Coefficients:
# (Intercept)            x            z  
#     0.51526      0.12572     -0.03936 

xs = seq(-8,8,0.001)
lines(xs, mod$coefficients[1] + mod$coefficients[2] * xs ,
     col = 1, lty = 1)
lines(xs, mod$coefficients[1] + mod$coefficients[2] * xs + mod$coefficients[3] ,
     col = 2, lty = 1)


### legend
legend(-8.2,0.95, c("data points z=0", "data points z=1", 
                      "true model z=0","true model z=1",
                      "linear OLS fit z=0", "linear OLS fit z=1"),
       pch = c(21,21,NA,NA,NA,NA),
       lwd = c(NA,NA,2,2,1,1),
       lty = c(NA,NA,2,2,1,1),
       col = c(1,1,1,2,1,2),
       pt.bg = c(0,2,NA,NA,NA,NA),
       cex = 0.6)

### glm model fit
mod2 = glm(y~x+z, family = binomial)
mod2 
## result
# Coefficients:
# (Intercept)            x            z  
#     0.06146      1.03038      1.28454 
$\endgroup$
3
  • 2
    $\begingroup$ A search on this site today shows 534 posts related to model misspecification. The tag info page has links for further reading. $\endgroup$
    – EdM
    May 12 at 15:18
  • $\begingroup$ EdM +1 so it seems a specific type of specification error. $\endgroup$ May 12 at 15:40
  • $\begingroup$ That's how I'd describe it. But there are so many types of specification errors. I don't know whether this particular type has a particular name. $\endgroup$
    – EdM
    May 12 at 16:01

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.