Fun question.
In the first question, you are labelling objects as either A or B, making a definitive choice. If you do this for a sequence of objects, you can construct what is known as a confusion matrix
Actually A Actually B
Assigned A a b
Assigned B c d
From this matrix, we can compute a whole host of metrics. Accuracy is probably the one people think of most often, but sensitivity and specificity are two other measures of how well one can guess.
In the second question, you would be assigning probabilities (p% this object is class A, q% this object is class B, etc). In that case, calibration of your probabilities is a measure you would want to examine. In brief, say you had 20 objects which you p[predicted all have 95% probability of being class A. If you are well calibrated for predicting the probability of class membership, then 19 of those 20 should actually belong to class A (that is 95% of the 20 actually belong to class A).
As Arya mentions, these are quite standard topics in the realm of classification and probabilistic modelling respectively. I will leave you to understand them further.