Intepretation of a probabilistic classifier's result

Question

I understand that a probabilistic classifier predicts the probability distribution $P(C|X)$. However I do not understand what $P(C|X)$ means.

Is it "$P(C=c|x)$ the probability of belonging to class $c$ given the input $x \in X$"? If so then what is the sample sample?

I have done some researches and the term "calibration" comes up and it says that "$P(C=c|x) = p$ means that the $p$ is the probability of being correct for all inputs with the same output $p$". Is this the definition of the prediction or calibration is just a "metric" we are measuring our prediction?

Answer 1

Below is the population distribution of throwing two dices and to illustrate how to calculate the conditional probability.

In terms of using conditional probability to measure the accuracy of prediction/classification. In other words, it can be called Calibration - refers to the agreement between observed outcomes and predictions:

Assuming you are building two models to recognize the pictures of dogs, Your testing data is 5000 pictures of dogs and cats.

P(predict from model 1 = dog| testing = dog) = 0.99

P(predict from model 1 = dog | testing = cat) = 0.88

---

P(predict from model 2 = dog | testing = dog) = 0.89

P(predict from model 2 = dog | testing = cat) = 0.10

So, model 1 is overfitted and we prefer model 2. It means the P(predict|testing) can be used to measure the accuracy of the model.