Return to Question

	7	edited tags \| link	edited Mar 2 at 9:37 amoeba 39.8k10138201
forecasting decision-theory calibration brier-score logarithmic-scoring-rule scoring-rules

		Notice removed Canonical answer required by Christian	occurred Jan 10 at 10:08

		Bounty Ended with Gumeo's answer chosen by Christian	occurred Jan 10 at 10:08

		Notice added Canonical answer required by Christian	occurred Jan 3 at 10:26

		Bounty Started worth 50 reputation by Christian	occurred Jan 3 at 10:26

	6	edited tags \| link	edited Dec 28 '16 at 11:24 Christian 5211626
forecasting prediction decision-theory calibration brier-score logarithmic-scoring-rule

	5	deleted 7 characters in body source \| link	edited Dec 28 '16 at 11:15 Christian 5211626
inline side-by-side side-by-side markdown I program and do test-driven development. After I made a change in my code I run my tests. Sometimes they succeed and sometimes they fail. Before I run a test I write down a number from 0.01 to 0.99 for my credence that the test will succeed. I want to know whether I'm improving in predicting whether my test will succeed or fail. It would also be nice if I can track whether I'm better at predicting whether the test will succeed on Mondays or on Fridays. If my ability to predict test success correlates with other metrics I track, I want to know. That leaves me with the task of choosing the right metric. In Superforcasting Philip Tetlock proposes to use the Brier scoring rulescore to measure how well experts are calibrated. Another metric that has been proposed in the literature is the Logarithmic scoring rule. There are also other possible candidates. How do I decide which metric to use? Is there an argument for favoring one scoring rule over the others? I program and do test-driven development. After I made a change in my code I run my tests. Sometimes they succeed and sometimes they fail. Before I run a test I write down a number from 0.01 to 0.99 for my credence that the test will succeed. I want to know whether I'm improving in predicting whether my test will succeed or fail. It would also be nice if I can track whether I'm better at predicting whether the test will succeed on Mondays or on Fridays. If my ability to predict test success correlates with other metrics I track, I want to know. That leaves me with the task of choosing the right metric. In Superforcasting Philip Tetlock proposes to use the Brier scoring rule to measure how well experts are calibrated. Another metric that has been proposed in the literature is the Logarithmic scoring rule. There are also other possible candidates. How do I decide which metric to use? Is there an argument for favoring one scoring rule over the others? I program and do test-driven development. After I made a change in my code I run my tests. Sometimes they succeed and sometimes they fail. Before I run a test I write down a number from 0.01 to 0.99 for my credence that the test will succeed. I want to know whether I'm improving in predicting whether my test will succeed or fail. It would also be nice if I can track whether I'm better at predicting whether the test will succeed on Mondays or on Fridays. If my ability to predict test success correlates with other metrics I track, I want to know. That leaves me with the task of choosing the right metric. In Superforcasting Philip Tetlock proposes to use the Brier score to measure how well experts are calibrated. Another metric that has been proposed in the literature is the Logarithmic scoring rule. There are also other possible candidates. How do I decide which metric to use? Is there an argument for favoring one scoring rule over the others?

	4	edited tags \| link	edited Dec 28 '16 at 11:05 Christian 5211626


	3	edited tags \| link	edited Dec 28 '16 at 10:54 Christian 5211626


		Tweeted twitter.com/StackStats/status/813965845449863168	occurred Dec 28 '16 at 4:32

	2	added 8 characters in body source \| link	edited Dec 27 '16 at 22:42 Christian 5211626


	1	source \| link	asked Dec 27 '16 at 22:36 Christian 5211626

current community

your communities

more stack exchange communities

Return to Question