Scoring rules are used to assess the accuracy of predicted probabilities, or more generally of predictive densities. Examples of scoring rules include the logarithmic, Brier, spherical, ranked probability and the Dawid-Sebastiani score and the predictive deviance.
Assume we are forecasting or predicting for multiple future realizations $k=1, \dots, K$. At each $k$, we observe a new realization $y_k$, which is distributed according to some unknown distribution, $y_k\sim f_k$. We have probabilistic or density forecasts or predictions $\hat{f}_k$.
A scoring rule is a function $s$ of the observed outcome $y$ and the density forecast $\hat{f}$. We will typically evaluate it on concrete realizations by taking averages,
$$\frac{1}{K}\sum_{k=1}^K s(y_k, \hat{f}_k),$$
and compare this value between different models that yield different $\hat{f}_k$s.
A scoring rule $s$ is proper if it is minimized in expectation if $\hat{f}_k=f_k$ for all $k$. It is strictly proper if it is minimized in expectation only if $\hat{f}_k=f_k$ for all $k$. Scoring by misclassification rate is not a proper score rule. Common scoring rules include the Brier score, the logarithmic score, and the spherical score.
More information can be found in Gneiting & Katzfuss (2014, Annual Review of Statistics and Its Application), Gneiting & Raftery (2007), Journal of the American Statistical Association, and Czado, Gneiting & Held (2009, Biometrics) specifically for count data. Merkle & Steyvers (2013, Decision Analysis) discuss how different scoring rules hang together, and how to choose one. Gneiting & Ranjan (2011, JASA) discuss weighted scoring rules for when we want to emphasize regions of interest, like the center or tails of the probability density, without losing propriety. Why is LogLoss preferred over other proper scoring rules? specifically presents arguments for and against the log and the Brier score.
Note that the above definition is correct for "negatively oriented" scoring rules (lower is better). The opposite convention of "positively oriented" scoring rules (higher is better) is also common in the literature - frequently by just using a minus sign in the definition. Unfortunately, not all papers explicitly note which convention they are following.
