0
$\begingroup$

In a model run of elastic net logistic regression, I encountered a very counterintuitive coefficient. I have looked into the data, model and script, but, I still cannot seem to wrap my head around the counter-intuitiveness I see regarding the dependent and independent variable. Initially, the V7 should be negative, as it is significantly lower in the dependent variable where the outcome is 1 compared to the outcome of 0, see graph.

graph

Further, the descriptive statistics are:

               Value 0          Value 1
count    749304.000000       402.000000
mean          2.762876         1.618396
std           3.672386         2.488794
min           0.000000         0.000000
25%           0.306000         0.001500
50%           1.662000         0.638250
75%           3.901500         2.338500
max         223.084500        17.217000

But, I end up with coefficients that show the following, here, one should keep an eye on variable number 7 (V7), which I am talking about.

(Intercept) -3.096141e+01
V1           1.436113e-03
V2          -1.774919e-01
V3          -5.586214e-04
V4          -1.763915e-03
V5           6.817795e-03
V6           3.986299e-02
**V7         3.085392e-02**
V8          -1.117509e-02
V9           6.917977e-02
  1. Why do I see that coefficient V7 is positive when it clearly is smaller in cases of 1 than cases of 0 in the dependent variable?
  2. Do I misinterpret the results of my elastic net regression? I doubt it, as the other variables are intuitively correct?

The script is below:

library(readr)
library(caret)
library(tidyverse)
library(glmnet)
library(ROCR)
library(pROC)
library(doParallel)
registerDoParallel(4, cores = 4)
set.seed(123)
df <- read_csv("df.csv")
View(df)
training.samples <- df$V10 %>% createDataPartition(p = 0.8, list = FALSE)
train <- df[training.samples, ]
test <- df[-training.samples, ]
x.train <- data.frame(train[, names(train) != "V10"])
x.train <- data.matrix(x.train)
y.train <- train$fire
x.test <- data.frame(test[, names(test) != "V10"])
x.test <- data.matrix(x.test)
y.test <- test$fire
nFolds <- 10
foldid <- sample(rep(seq(nFolds), length.out = nrow(train)))
list.of.fits <- list()
for (i in 0:10){
    fit.name <- paste0("alpha", i/10) 
    list.of.fits[[fit.name]] <- cv.glmnet(x.train, y.train, type.measure = "deviance", alpha = i/10, family = "binomial", nfolds = nFolds, foldid = foldid, parallel = TRUE)
}
coef <- coef(list.of.fits[[fit.name]], s = list.of.fits[[fit.name]]$lambda.min)
coef
Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ this is hard to see. Could you log10 scale your y-axis? Would you mind moving to ggplot and putting a jitter behind the boxplot? $\endgroup$ – EngrStudent Sep 21 '20 at 13:48
  • 1
    $\begingroup$ Remember that the coefficients are biased, so you're chasing a "wrong" answer in exchange for getting less variability in the coefficient estimates. $\endgroup$ – Dave Sep 21 '20 at 13:57
  • $\begingroup$ Does this problem persist if you consider a model with V7 as the only explicative variable? $\endgroup$ – David Sep 21 '20 at 14:06
  • $\begingroup$ @EngrStudent does the graph help? Thank you. $\endgroup$ – Thomas Sep 21 '20 at 15:40
  • 1
    $\begingroup$ This is some amazingly imbalanced data. You have O(million) samples of case "0" and O(hundred) of case "1". That is 3 decades of imbalance, which can be really large and have its own pathologies. You then do 10-fold CV on it. $\endgroup$ – EngrStudent Sep 21 '20 at 18:17
1
$\begingroup$

It is not uncommon for coefficients to change sign when other terms are added to the model, e.g. when x7 is the only term in the model then the coefficient will be negative as you expect, but as you add more terms (either through a stepwise approach, or relaxing the penalties/constraints in the elasticnet) the coefficient on x7 can easily change sign due to relationships between x7, the other predictors, and the response.

See this answer: Binary Logistic Regression: Direction of B's different in multiple than in bivariate cases for one example/explanation that may make this more clear.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Would this be (or could this be) an example of Simpson's paradox? $\endgroup$ – Dave Sep 21 '20 at 16:06
  • $\begingroup$ @Dave, for the broader definition of Simpson's paradox, Yes. $\endgroup$ – Greg Snow Sep 21 '20 at 19:57
  • $\begingroup$ But, shouldn't the Elastic Net regression namely not 'care' about the relationship between the variables that are put in the model? $\endgroup$ – Thomas Sep 22 '20 at 7:25
  • $\begingroup$ @Thomas, elastic net regression (and ridge regression and lasso) just bias the parameters towards 0 (through penalty or constraint), This reduces the high variability due to high multicolinearity, but the individual coefficients are still adjusted for each other within the constraint, so the relationships between variables will still affect the coefficients (at least until the constraint/penalty is severe enough to leave at most one non-zero coefficient). $\endgroup$ – Greg Snow Sep 22 '20 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.