i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. clearly they are very close! how can i assess if they are statistically different?

i have used mcnemars test:

 from mlxtend.evaluate import mcnemar_table
    from mlxtend.evaluate import mcnemar
    lgbm_pred = second_best.predict(x_test)
    xg_pred = chosen_model.predict(x_test)
    tb = mcnemar_table(y_target=y_test, 
    chi2, p = mcnemar(ary=tb, corrected=True)
    print('chi-squared:', chi2)
    print('p-value:', p)

output is: chi-squared: 2.25 p-value: 0.13361440253771584, so i would not reject null that models peformance are equal. (hopefully i am using this correctly so pls let me know if i am not.)

i have seen some threads on using 'permuation tests' etc.. but i am unsure how to interpret these and also i thought these tests were only if you have small sample size which i don't.


basically how can i assess which classifier being better? when in an ideal world i would want a classifer which is able to predict has cancer. can i compare precisions of model to model? what is best approach.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.