Why would this simple metric be wrong as a classifier's measure quality? I have the outputs from two classifiers for a given test set, each of the two provided by a different party, and treated as black boxes. I would like to make a comparative evaluation of them, to give an idea of which one is better on the task.
What would be the chief reasons why the following function would be dead wrong, as a measure of classification quality that might be used to juxtapose the performance of the two classifiers?
For all data points in the test set: apply a function that penalizes the classifier's confidence for each datum belonging to the class, with its distance from 1 (for those datums that are in the class) and penalize with its distance from 0 for those datums that are out of the class. Then we average or otherwise normalize.
 A: I'm not sure I know what "dead wrong" means, but your loss function has the flaw that it decouples from accuracy.  Imagine a classifier that is 100% accurate but with low confidence, say it always predicts the correct class with probability of 51%.  On any point, the classifier error is .49 and if there are N observations, your loss function takes the value $N*.49$ .  Now imagine a classifier that predicts 'with certainty', that is gives probability as 100% for every guess, but is only 80% accurate. If there are N observations, then the expected value of the loss function is $N*(.8*0 + .2*1) = .2*N $, which is much lower.  I think for most data sets, this is not desirable.
This can be made more exact. If your model is $ A \geq .5 $ accurate and one assumes that the model predicts with constant probability x (so $ .5 < x < 1$),
then the expected loss is $ A*(1-x) + (1-A)*x = A + x(1-2A)  $.  If you're model is more than 50% accurate, then $1-2A$ is negative, so the expected loss is minimized by the smallest  positive x which is $ x = .5 + \epsilon $ since   $ .5 < x < 1$
A: What you described is a scoring rule: let's call it a linear scoring rule. When compared to other popular scoring rules, one of its disadvantages is that it's not a proper scoring rule, i.e. the best reward is not achieved by the reporting of the *true* probability distribution.
You can read more on proper scoring rules at wikipedia for example, and there's also this good reference [1]: 

Essentially, the linear score encourages overprediction at the modes
  of an assessor's true predictive density.

Having said that, linear scoring rules do find some applications.

[1] Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.
