0
$\begingroup$

I have tried GridSearchCV and BayesSearchCV for tuning my lightGBM algorithm (for binary classification).

I have used 10 iterations and I have indicated scoring ="roc_auc"

In the first iteration, I have got:

best score (e.g :0.71...) 
and best param (e.g: max-depth: 10 , learning-rate: 0.17..., num-leave:175, n-estimators: 176, ....)

In the 10 iteration, I have got :

best score (e.g :0.72...) 
and best param (e.g: max-depth: 9 , learning-rate: 0.19..., num-leave:168, n-estimators: 172, ....)

Then I tried to train my LightGBM classifier with the last param (which supposed that it get the best score!!). I have got :

AUC : (0.7541.., 0.6467..)
Accuracy: 0.7338..
RMSE: 0.5216..

Then because I had some curiosity I have tried to train my classifier with (best param) of the First iteration (which considerate as worst score) ! . I had surprised by the result getting:

AUC : (0.7545., 0.6592..)
Accuracy: 0.7332..
RMSE: 0.5152..

Because I have fixed previosly scoring by roc-auc I should get AUC in the 10 iteration better than the first iteration but I have got the contraire.

I have supposed that it concederate AUC train of 10 iteration 0.7541 is better then 0.7545 of the 1st because of the overfitting but when I tried to check the 3rd and the 5th iteration I get on the 3rd 0.7532 and in the 5th 0.7548.

So I don't know what the best score in this algorithm mean . And why we should if it get values as the described situation . (I had tried then many times with other tuning parametres i get the same case. I dont know where is the problem exactly.

My sourse code :

# X and y data are splited to 10 fold Cross- validation


import lightgbm as lgb
from skopt import BayesSearchCV
from sklearn.model_selection import StratifiedKFold

ITERATIONS =10
lghtboost = lgb.LGBMClassifier(silent=False, n_jobs=7, verbose=0,  metric=['roc_auc'])

param_grid ={'max_depth': [6, 15], 'learning_rate': [0.1,0.2], 'num_leaves': [150, 210], 
                       'n_estimators': [100,180], 'min_child_samples': [5], 'colsample_bytree': [1] }

bayes_cv_tuner_lgh = BayesSearchCV(
        lghtboost,
        param_grid,
        n_iter=ITERATIONS,
        scoring = 'roc_auc',
        cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0),
        refit = True,
        random_state = 0,
        verbose=0,
        n_jobs=7 
        )

def status_print(optim_result):
     print('Best ROC-AUC: {}'.format(bayes_cv_tuner_lgh.best_score_))
     print('params = {}'.format(bayes_cv_tuner_lgh.best_params_))

result_Lightboost = bayes_cv_tuner_lgh.fit(X_train.values, y_train.values, callback=status_print)



d_train = lgb.Dataset(X_train, label=y_train)

## I press the last best parameter:
params = {max-depth: 9 , learning-rate: 0.19..., num-leave:168, n-estimators: 172, ....}

model_lgh = lgb.train(params, d_train)
y_pred = model_lgh.predict(X_test)

def auc_LGBM(m, train, test): 
    return (metrics.roc_auc_score(y_train,m.predict(train)), metrics.roc_auc_score(y_test,m.predict(test)))

print('AUC : ', auc_LGBM(model_lgh, X_train, X_test))
print("Accuracy : ", metrics.accuracy_score(y_test, y_pred.round()))
print("RMSE : ", np.sqrt(metrics.mean_squared_error(y_test, y_pred.round())))
$\endgroup$
  • $\begingroup$ Without knowing anything in particular about your problem (which could make everything I say invalid) I guess from experience that you have already reached a level of performance that you cannot improve any further using hyperparameter tuning... I.e. you get AUCs that are the same up to the third digit... it seems to me as if the deviation in the fourth digit is a minor random effect and not something that is really caused by different hyperparameters. I would recommend to do some more feature engineering, useing lagged features, etc in order to improve performance and leave hyperparameter ... $\endgroup$ – Fabian Werner Mar 19 '19 at 13:31
  • $\begingroup$ tuning to the very end of the model-building-exploratory phase as a 'last' possibility of improving the performance (however, I have never seen that a well tuned model and a super mega hyper duper fine tuned model give significally different performances... here, significally depends on what you want to do though: If you want to win a Kaggle competition then you must tune hyperparams but in the 'real' world I have never seen an advantage of Bayesian Optimization over, say, RandomSearch). $\endgroup$ – Fabian Werner Mar 19 '19 at 13:33
  • $\begingroup$ Thank fabian for your replay, concerning your answers 'My algorithm has reached a level of performance that I cannot improve' : (depend on what I understand) If it is the case, normally when I tried to calculate AUC metrics after training and predicting model based on the last best_param (which is the parameter of the 10th iteration I should get bigger AUC score that the auc score getted on the 1st iteration. $\endgroup$ – Nirmine Mar 19 '19 at 15:00
  • $\begingroup$ I want to know something: what is the relationship between the best_score returned by bayesSearchCV and the auc value after training and predicting model. $\endgroup$ – Nirmine Mar 19 '19 at 15:03
  • $\begingroup$ I want to know why when I try to fit my algorithm with the result returned of 1st iteration, I get the best AUC, I have used the best_param of the 2nd iteration and I found worst AUC value than 1st iteration. I tried the best_param of the 3rd iteration, I find better AUC than 2nd iteration but worst AUC value than 1st iteration without any logic. $\endgroup$ – Nirmine Mar 19 '19 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.