I have a classifier that gives its decisions as probability estimations: for each datum it returns a set of probabilities $p_j$ for each known class $j$: $p_j(\vec{x})=P(c_{real}=j|\vec{x})$.
I have a training dataset, on which I trained the classifier, and I have a test dataset to test it. In both datasets I have the data in form of $(\vec{x_i}, y_i)$, where $\vec{x_i}$ is the feature vector of each datum, and $y_i$ is the right human-assigned class of each datum (each datum can have only one right class assigned).
I can easily evaluate the quality of classification with PR and ROC curves, but I also want to evaluate the quality of probability estimations, that makes my classifier.
Now I'm using the next way:
1) I distribute all the $p_j$ in 10 bins:
$(0 \dots 0.1), (0.1 \ldots 0.2), \ldots , (0.9 \ldots 1.0)$.
2) Then for each bin I look, how many decisions fall into each bin, and how many of them are right decisions: $P_{bin}=\frac{right-decisions-in-bin}{total-decisions-in-bin}$
3) Having measured probability curve, I just look at its sum of squared deviations from $f(x)=x$ in each point and try to have a classifier when such sum is minimal.
But I do think that it not the best way of such an evaluation of probability estimation. Cannot you advice me something more reliable? Maybe, there are some standard techniques?
P.S. Some probability curves, measured in my way:
The text marks on the plot show (#right decisions in bin / #decisions in bin) for one of the measured probability functions.