Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm looking for a metric to evaluate my classification model. I have 3 differents class (0,1,2). And I want to get the average probability of the good label. For example, if my ml model get me those probas : [[0.5, 0.25, 0.25], [0.1, 0.8, 0.1], [0.7, 0.2, 0.1]] And the true labels are [0,1,0], then the evaluation result should be (0.5+0.8+0.7)/3 = 0.66.
Does this metric has a name ?
Yup! It has a name, and it makes sense to care about. It's called the expected accuracy—and alternatively, one minus the expected error rate.
The idea is that you pretend to pick a class randomly from the posterior distribution instead of picking the most probable class. (That is—stochastic decoding instead of MAP decoding.) If the probability of the correct class is 0.8, you'll still pick this 80% of the time, so you get 80% credit.
