Edit - Cross Validated

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome all constructive edits, but please make them substantial. Avoid trivial edits unless absolutely necessary.

How do I choose the best metric to measure my calibration?

I program and do test-driven development. After I made a change in my code I run my tests. Sometimes they succeed and sometimes they fail. Before I run a test I write down a number from 0.01 to 0.99 for my credence that the test will succeed.

I want to know whether I'm improving in predicting whether my test will succeed or fail. It would also be nice if I can track whether I'm better at predicting whether the test will succeed on Mondays or on Fridays. If my ability to predict test success correlates with other metrics I track, I want to know.

That leaves me with the task of choosing the right metric. In Superforcasting Philip Tetlock proposes to use the Brier score to measure how well experts are calibrated. Another metric that has been proposed in the literature is the Logarithmic scoring rule. There are also other possible candidates.

How do I decide which metric to use? Is there an argument for favoring one scoring rule over the others?

Answer

Although this is far from an answer and more of a reference, it might be a good idea to check [Steyerberg E - Epidemiology 2012][1].

In this article Steyerberg and colleagues explain different ways to check prediction model performance for models with binary outcomes (succes or failure). Calibration is just one of these measures. Depending on whether you want to have an accurate **probability**, **accurate classification**, or **accurate reclassification** you might want to use different measures of model performance.
Even though this manuscript concerns models to be used in biomedical research I feel they could be applicable to other situations (yours) as well.

More specific to your situation, calibration metrics are really difficult to interpret because they summarize (i.e. average) the calibration over the entire range of possible predictions. Consequently, you might have a good calibration summary score, while your predictions were off in an important range of predicted probabilities (e.g. you might have a low (=good) brier score, while the prediction for succes is off in above or below a certain predicted probability) or vice versa (a poor summary score, while predictions are well-calibrated in the critical area).
I would therefore suggest you think about whether such a critical range of predicted succes probability exists in your case. If so, use the appropriate measures (e.g. reclassification indices). If not (meaning you are interested in overall calibration), use brier, or check the intercept and slopes of your calibration plot (see Steyerberg article).

To conclude, any one of the calibration summary measures require your first step to plot your predicted probabilities versus the observed probability (see Outlier's answer for example how to). Next, the summary measure can be calculated, but the choice of summary measure should reflect the goal of predicting succes of failure in the first place.

[1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/

Edit Summary

cancel

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

► complete the sentence: my question is about...

► use tags that describe things or concepts that are essential, not incidental to your question

► favor using existing popular tags

► read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

► combine multiple words into single-words with hyphens (e.g. machine-learning), up to a maximum of 35 characters

► creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »

current community

your communities

more stack exchange communities

How do I choose the best metric to measure my calibration?

Answer

How to Edit

How to Format

How to Tag