I'm working on an online model scoring framework, my goal is to be able to understand if my model's predictive performance is degrading week-over-week. I have a classification model (trained on binary target data) which produces probabilities, and I am most interested in assessing the accuracy of those probabilities. For model selection and tuning, i'm using log loss, but I understand that log loss can't be used to compare performance across different datasets (different weeks of predictions). Is there a best practice metric that I can use to effectively compare the accuracy of predicted probabilities, across different datasets?

I'm working on an online model scoring framework, my goal is to be able to understand if my model's predictive performance is degrading week-over-week. I have a classification model (trained on binary target data) which produces probabilities, and I am most interested in assessing the accuracy of those probabilities. For model selection and tuning, i'm using log loss, but I understand that log loss can't be used to compare performance across different datasets (different weeks of predictions). Is there a best practice metric that I can use to effectively compare the accuracy of predicted probabilities, across different datasets? More explicitly, what are appropriate metrics to monitor predicted probability quality over time?