Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy loss (logistic regression and deep models using a final softmax layer).
Minimizing crossentropy loss is equivalent to maximum likelihood estimation of the regression coefficients and thus to maximum likelihood estimation of the conditional probabilities. (Since the probabilities are a decent function of the regression coefficients, the MLE of the probabilities is that same decent function of the regression coefficient MLE.)
Consequently, yes, the models seek out something like $P(Y\vert X)$, but they go beyond that. When you have multiple classes (more than just two), these models are supposed to seek out the true probabilities of all classes, not just the one with the highest probability.
Neural networks are known for giving poor probability predictions, however, so I will leave a link to a related question of mine about calibration of the probabilities of the non-dominant classes.
