2
$\begingroup$

I am trying to predict binary outcome (Response) with a single feature (V1) using scikit-learn implementation of Logistic Regression (default parameters).

V1: [2.56, 1.12, 1.38, 1.38, 1.25, 1.28, 0.92, 1.51, 2.23, 1.91, 2. , 1.31, 1.41, 1.51, 1.35, 1.12, 1.68, 1.94, 1.61, 2.56, 2.23, 1.54, 1.41, 2.04, 1.18, 1.38, 1.48, 1.54, 1.22, 1.08, 1.77, 2.17, 1.58, 1.38, 1.35, 1.38, 1.35, 1.87, 1.51, 1.77, 1.28, 1.48, 1.74, 1.81, 1.84, 1.84, 1.84, 1.71, 1.84, 1.91, 1.31, 2. , 2.04, 1.35, 1.71, 1.51, 1.31, 1.54, 1.51, 1.38, 1.77, 1.38, 1.12, 1.61, 1.41, 1.68, 1.84, 1.81, 2. , 2.27, 1.05, 2.07, 2. , 1.12, 1.91, 1.97, 1.81, 2.17, 1.28, 1.38, 1.81, 1.48, 1.48, 1.64, 1.97, 2.23, 1.35, 2.2 , 1.77, 1.38, 1.81, 1.58, 1.87, 1.61, 1.58, 1.84, 1.31, 0.92, 1.84, 1.61, 1.18, 1.61, 1.71, 1.31, 1.41, 2.1 , 1.41, 1.81, 1.48, 1.74, 1.41, 1.84, 1.35, 1.54, 1.71, 1.68, 2.1 , 1.61, 1.08, 1.77, 1.61, 1.84, 2.23, 1.91, 1.77, 1.71, 1.68, 2.46, 2. , 2. , 1.97, 2.5 , 2.3 , 2.04, 2.04, 1.94, 1.54, 2.66, 2.04, 1.51, 2.04, 1.91, 2.14, 1.58, 1.77, 1.94, 0.13, 0.16]
Response:[0., 0., 0., 0., 0., 1., 1., 0., 1., 0., 0., 1., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0.]

I used k-fold cross-validation with 30 repetitions. The mean test AUC I got is 50%, which means the model has no skill to distinguish the two classes. However, when I plot ROC at different thresholds of feature V1, i get AUC of 0.35, showing a negative correlation with the outcome variable. See figure below

ROC_AUC plot for Response at different thresholds of V1

Also results from univariate statistical tests give significant p-value (0.02). I used glm function from R for univariate test as I like it better for statiscal anlaysis

dm_data <- import("Myexcel.xlsx")

glm.fit <- glm(Response ~ ., data = dm_data, family = binomial("logit"), maxit=100)

[Output]

Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)   0.4229     0.9341   0.453   0.6508  
V1           -1.3228     0.5868  -2.254   0.0242 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 127.86  on 147  degrees of freedom
Residual deviance: 122.54  on 146  degrees of freedom
AIC: 126.54

Number of Fisher Scoring iterations: 4

Thus I assume that this variable is atleast a moderate predictor of the outcome. but median AUC i get from Logisticregression doesnot support my analysis. My implementation is as follow

dataset=pd.read_excel('Myexcel.xlsx')
np_dataset=dataset.values
X= np_dataset[:,1]
y=np_dataset[:,0]
X=X.reshape(-1,1) # reshaped as there is only feature

#Gridsearch for best params

grid_param = {
        'penalty':['l1', 'l2', 'elasticnet' 'none'],
        'solver' : ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'],
        'C': [0.001,0.01,0.1,1.0]}
gsc = GridSearchCV(estimator=LogisticRegression(),
                     param_grid=grid_param,
                     scoring='roc_auc',
                     cv=3,
                     n_jobs=-1)
grid_result = gsc.fit(X, y)
best_params = grid_result.best_params_

kfold  = RepeatedStratifiedKFold(n_splits=3, n_repeats=30,random_state=1)
ROC_test=[]

for train, test in kfold.split(X,y):
    model = LogisticRegression(C=best_params['C'], penalty=best_params['penalty'], solver=best_params['solver'])
    model.fit(X[train], y[train])
    yhat_test= model.predict_proba(X[test])
    auc_test= roc_auc_score(y[test], yhat_test)
    ROC_test.append(auc_test)
print('Test AUC Median: %.2f' %np.median(ROC_test)*100))

[Output]
Test AUC Median: 50.00%

Can anyone please help me to understand the reason for this 0.5 auc of logisticregression model. Thanks

$\endgroup$
4
  • $\begingroup$ Hello -- two things. 1. Please show output from one of your logistic regressions and one of your univariate tests. 2. "[W]hen I plot ROC at different thresholds of feature V1" - you should get multiple sets of AUC results, yet you say simply "i get AUC of 0.35." Please clarify. $\endgroup$
    – rolando2
    Jun 1, 2020 at 11:56
  • $\begingroup$ Hi, Sorry for incomplete information, I have edited my question by adding results of univariate analysis that also uses logistic regression and computes p-value which came out to be 0.02 (less than alpha 0.05) for feature V1. about AUC may be i did not write it clearly but the way you get ROC plot for single feature is always a single plot where you first arrange values of feature and compute true postive and false positive rate for each value of feature (which is refered as threshold). So i think it should be as single plot not multiple. $\endgroup$ Jun 1, 2020 at 12:33
  • $\begingroup$ The 2nd part of my comment must have been a mistake. And I don't know how all your code works. But is it possible the negative sign of your coefficient, or the values of your Y, are somehow prompting a reversal, and that your AUC of (.50-.15) should be re-expressed as (.50+.15)? $\endgroup$
    – rolando2
    Jun 1, 2020 at 12:49
  • $\begingroup$ the coefficient for V1 is definetly negative becuase if i look closer to data i see that smaller the value of V1 greater is occurrence of event 1. Thus there is a weak correlation that previously was not captured in my CV folds because of highly imbalanced data. Now I balanced the data by enabling 'class_weight' parameter in logisticregression and results are pretty acceptable. I will aslo try to use SMOTE method from 'imblearn' libarary to balance the data and see if it does any good to final AUC. $\endgroup$ Jun 1, 2020 at 13:02

2 Answers 2

4
$\begingroup$

Below is the plot of response vs V1, and I don't see a good predictor here:

enter image description here

More importantly, if you inspect your data better, you'll see that most of the class 1 samples have x values belonging class 0 as well, e.g. V1 = 1.28 has 2 class-0 responses, and 1 class-1 response.

$\endgroup$
1
  • $\begingroup$ Thank you @gunes. I also agree that V1 is not a very good predictor and classes are overlapping and highly imbalanced, but the negative correlation I see from AUC plot at different threshold values of V1 gives me an idea that there is a value of V1 which is somehow able to roughly seperate two classes e.g. V1 = 1.5, but logisticregression is unable to capture even a slight correlation and AUC stays 0.5 for all repititions. I also edit my code and added gridsearchcv for best parameters but nothing changed. $\endgroup$ Jun 1, 2020 at 8:08
0
$\begingroup$

I think I found solution. Better AUC can be achieved by adjusting for class imbalance. Previously number of samples from class 1 were too small in each repetition that model was unable to make any Prediction on it. I followed some steps from here optimizing logisticregression

adding class_weight='balanced' in model deals with class imbalance and it has improved results to median AUC of 0.58. Small addition in model parameters is as follow

model = LogisticRegression(C=best_params['C'], penalty=best_params['penalty'], solver=best_params['solver'],class_weight= 'balanced')

After fitting this model results are as follow:

[Output] 

Test AUC Median: 58.33%

Also I see that for all repetitions results are different and better than 0.5. AUC for few repetitions that is stored in list 'ROC_test' is:

ROC_test
[Output]
0.66, 0.511,0.60,0.54,0.59,0.65,0.63,0.49,0.63,0.59,0.58,0.60,0.54,0.75...

I am now happy with model performance as I am also not expecting much from overlapped values of two classes as mentioned by @gunes.

$\endgroup$
1
  • 1
    $\begingroup$ You can also try other models. I did a project last year where I had a binary classification problem with a massive class imbalance (about 95% and 5%). A boosted model in a PCA transformed predictor space ended up doing the best. As you have only one predictor, the PCA won't be applicable here :) $\endgroup$
    – Stochastic
    Jun 1, 2020 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.