0
$\begingroup$

all,

i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. clearly they are very close! how can i assess if they are statistically different?

i have used mcnemars test:

 from mlxtend.evaluate import mcnemar_table
    from mlxtend.evaluate import mcnemar
    
    lgbm_pred = second_best.predict(x_test)
    xg_pred = chosen_model.predict(x_test)
    
    tb = mcnemar_table(y_target=y_test, 
                       y_model1=lgbm_pred, 
                       y_model2=xg_pred)
    
    chi2, p = mcnemar(ary=tb, corrected=True)
    print('chi-squared:', chi2)
    print('p-value:', p)

output is: chi-squared: 2.25 p-value: 0.13361440253771584, so i would not reject null that models peformance are equal. (hopefully i am using this correctly so pls let me know if i am not.)

i have seen some threads on using 'permuation tests' etc.. but i am unsure how to interpret these and also i thought these tests were only if you have small sample size which i don't.

https://stackoverflow.com/questions/52373318/how-to-compare-roc-auc-scores-of-different-binary-classifiers-and-assess-statist

basically how can i assess which classifier being better? when in an ideal world i would want a classifer which is able to predict has cancer. can i compare precisions of model to model? what is best approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.