Scoring a model with "distance from truth" Scoring classification model performance often seems somewhat abstract (looking at you AUC scores...). There's always accuracy score, which has the advantage of being nice and easy to comprehend and which is great for explaining how well the model will work to someone else (like say, the people who are actually going to use the predictions it makes). I intuitively expect there to be a common similar method for probability predictions, for example a simple "average distance from truth" along the lines of:
| Truth | Prediction | Score |
| ----- | ---------- | ----- |
|   1   |     0.97   |  0.03 | 
|   0   |     0.35   |  0.35 |
|   1   |     0.76   |  0.24 |
|   0   |     0.42   |  0.42 |

With the score for the model as a whole being taken as the average of those scores; 0.26 in this case. That's pretty easy to manually do, but it surprises me that a) this isn't a common scoring metric and b) there doesn't seem to be any in-built methods in the scikit-learn api. 
So my question is this: is "average distance from truth" a useful scoring metric and if the answer is no, why not?
 A: The metric you describe is in fact very common: It's mean absolute error, or MAE. In scikit learn you can find it in the metrics submodule.
Usually it's used for regression tasks, not for classification, thus you might not have encountered it. Still, when it does get used to compare classification algorithms there are certain caveats, for example:


*

*For example, it has similar problems like accuracy when used with unbalanced datasets in that it will produce high scores for algorithms that just predict the majority class (and thus are not useful).

*The MAE doesn't tell you if your classifier better at predicting positives or negatives.


So to answer your question: It is a common, useful scoring metric, but less often used for classifiers (more common for regressors).
A: In addition to @Denwid's answer:

*

*Switching from MAE to MSE (mean squared error) will give you a proper scoring rule.


*You can then take its root (=> root mean squared error) to get a figure of merit in the original predicted unit for easier interpretation.


*The problems @Denwid refers to with unbalanced data and not giving the information whether the loss stems from false positives or false negatives has less to do with the choice of the loss function (0/1 loss for accuracy, mean absolute error or mean squared error): it is a concequence of "throwing" a loss function onto your whole data set - which will be problematic even for seemingly harmless figures of merit like total accuracy unless you make sure the relative class frequencies match those of your application scenario.
But: You can use MSE loss on subgroups of your data to derive MSE-figures of merit that are analogous to sensitivity, specificity, predictive values etc.
In case you are working in R, my package softclassval does provide such functions. We discussed details in our paper C. Beleites, R. Salzer and V. Sergo:  Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues,
Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22. AAM on arXiv:  1301.0264
