I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and choose the hyperparameters (n_estimators,learning_rate,etc.) with cross validation. My model by using training dataset shows 0.98 AUC and accuracy score of 0.880 from score function. I got a test dataset based on a published paper and use their test dataset to examine my model's accuracy. My model shows 0.73 AUC on their test dataset (in the article authors show 0.84 AUC for their test dataset) and 0.800 from score function on their test dataset (in the article authors show 0.76 accuracy score on their test dataset). By the way, authors used SVM instead of GBM to extract these results. Basically I could summarize my results as:

Training dataset: 0.98 AUC and 0.880 accuracy score

Test dataset: 0.73 AUC and 0.800 accuracy score (authors have 0.84 AUC and 0.76 accuracy score for test dataset)

My question is that: Can I sell my higher accuracy score in comparison on test dataset (0.800 > 0.76) despite my lower AUC? I appreciate any insight or suggestion or recommendation.


Browse other questions tagged or ask your own question.