How to compare probabilistic classifiers? Assume that we have a very long sequence (i.e. a list) of nominal-valued observations. For example:
A
A
C
B
...
B
A
B
C

We have also a corresponding sequence of predictions generated by a "predictor_1". Each prediction is given by the probabilities associated with the possible nominal observations. For example:
{A:0.5, B:0.1, C:0.4}
{A:0.6, B:0.0, C:0.4}
{A:0.7, B:0.2, C:0.1}
{A:0.2, B:0.2, C:0.6}
....
{A:0.2, B:0.2, C:0.6}
{A:0.7, B:0.2, C:0.1}
{A:0.5, B:0.1, C:0.4}
{A:0.6, B:0.0, C:0.4}

We also have another sequence of predictions (in the same format), generated by another predictor. For example:
{A:0.4, B:0.1, C:0.5}
{A:0.4, B:0.0, C:0.6}
{A:0.1, B:0.2, C:0.7}
{A:0.1, B:0.3, C:0.6}
....
{A:0.5, B:0.1, C:0.4}
{A:0.2, B:0.2, C:0.6}
{A:0.7, B:0.2, C:0.1}
{A:0.6, B:0.0, C:0.4}

Now, I want to determine what predictor is better. Or, in other words what predictions are "more informative".
The question is much more simple if our "predictors" give just one nominal value as a prediction (for example a predictor says that on the first step we expect to observe A then, on the second step, we expect B and so on). In this case we can use the percentage coincidences of predictions with the observations as a measure of the quality. But it is not clear to me what we should use in case of the above described "probabilistic" predictions. 
 A: With respect to probabilistic classifiers, there are multiple methods to evaluate models.  These include Root Mean Squared Error (RMSE), and Kullback-Leibler Divergence (KL Divergence), Kononenko and Bratko's Information Score (K&B), Information Reward (IR), and Bayesian Information Reward (BIR). Each have advantages and disadvantages that you should consider exploring.
To get you started, the simplest method for evaluating probability classifiers is RMSE.  The lower the value, the closer your model fits the predicted classes.  In the book, Evaluating Learning Algorithms: A Classification Perspective there is a brief example of the implementation by WEKA.
Here is the equation generalized for M possible classes.  Where N is the number of samples, $\hat{y}_{i}$ is the predicted probability and $y_{i}$ is the actual probability (i.e. 1 or 0).
$$RMSE = \sqrt{\frac{1}N\sum_{j=1}^{N}\sum_{i=1}^{M} \frac {(\hat{y}_{i}-{y}_{i})^2}M}$$
Let's go through an example to make it clear, here is a minimal table from your first predictor:
Sample  A_Pred  A_Actual    Diff^2/3    B_Predicted B_Actual    Diff^2/3    C_Predicted C_Actual    Diff^2/3    SqrErr
1       0.5     1           0.0833      0.1         0           0.0033      0.4         0           0.0533      0.14
2       0.6     1           0.0533      0           0           0           0.4         0           0.0533      0.1067
3       0.7     0           0.1633      0.2         0           0.0133      0.1         1           0.27        0.4467
4       0.2     0           0.0133      0.2         1           0.2133      0.6         0           0.12        0.3467
5       0.2     0           0.0133      0.2         1           0.2133      0.6         0           0.12        0.3467

Walking through this table, you will see the predicted probability for each class and its associated actual probability (it either is or is not that class, ergo 1 or 0).  Then you square the difference between actual and predicted for each class and divide by the number of classes (in your case 3).  This difference is then summed within each sample resulting in SqrErr (e.g. 0.0833+0.0033+0.0533 = 0.14).  Next, you take the sum of SqrError and divide it by the number of samples (i.e. N).
The sum of SqrErr = 1.3867
SqrErr/N = 0.2773
RMSE = sqrt(0.2773) = 0.5266

Not the best model but this was only using 5 samples too.  You should be able to apply this to your entire dataset and get an RMSE for each predictor/model.  A quick word of caution, you should be wary of overfitting your data and cross-validation is always recommended when training predictive models.
